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Abstract: 


Digital technologies have over the past few decades brought significant changes to the learning tools 
that are available to teachers and students. One of the reading interventions that has contributed in enhancing the 
reading ability of elementary school students is known as myON. This intervention helps students to easily 
identify books that they can use for independent reading. The primary aim of this study was to comprehend how 
elementary school students can obtain and use the digital book libraries through usability research. Further, this 
study was critical in helping to improve understanding on how myON can help students find relevant 
information about a book as well as their experiences while using myON website. Research findings indicated 
that approximately 97% of the participants showed success in task completion. This implies that the website was 
effective in enabling participants to successfully complete their tasks. Further, it was observed that the overall 
satisfaction score was above average, meaning that majority of the participants were satisfied with the website 
and its operations. There was a common consensus among the participants that myON was effective in enabling 
students to read a wide range of books online. 


Introduction 


The internet is widely used by students for academic purposes globally. With advancement in 
technology, it has become easier for students to employ the use of the internet as a research tool compared to the 
early years when the internet was only used for scientific research and military purposes (Syed, 2017). The 
internet has made significant progress to become the main tool of communication that is relied upon by 
everyone to convey various types of information (Radich, 2013). Despite the significant progress made in the 
expansion and access of the internet, there exits some negative aspects of internet use. For instance, there is 
limited regulation on the information that is published on the internet. In addition, information can be published 
over the internet by anyone, thus it can lead to the publication of biased and unreliable information which can be 
accessed by internet visitors (McDowell, Lytle & Rafail, 2016). Usability evaluation is critical as it emphasizes 
on how users can obtain and use a product to achieve their goals. Moreover, it can be used to describe the 
satisfaction of operators using the route. This paper sought to outline the various ways that can be used to 
enhance usability of digital book application to clients. 


Background 

Digital technologies have over the past few years dramatically changed the learning tools used by 
students and their teachers. The available evidence indicates that since 2007, there has been a significant 
increase in the number of applications and devices used for presenting digital book libraries (Biancarosa & 
Griffiths, 2012). The establishment of the digital reading programs was aimed at meeting the diverse learning 
needs and abilities of students. This is a critical aspect of successful reading programs. In the modern 
elementary classrooms, children possess diverse abilities. For example, some children know how to read upon 
entering the first grade while others do not have the basic pre-reading skills. For some children, English is not 


their first language, while others may suffer from learning disabilities. To address all these needs, digital reading 
programs have been developed to promote personalized reading. The development of the digital reading 
programs took into consideration the variation in students and consequently ensured that all students have access 
to resources that align with their level of ability (Brekhus, 2011). 

One of the key objectives of user experience (UX) research is to help identify the needs as well as 
expectations of users and to explore their interaction with the systems (Demir et al., 2017). Usability testing is a 
technique that helps determine the interaction of users with the product by collecting data on effectiveness, 
efficiency, the success rate of task completion, time taken to complete pre-defined tasks, and user satisfaction 
with the product (Demir, Karakaya & Tosun, 2012). Usability testing also helps in understanding how 
elementary school students can obtain and use the digital book libraries. As such, this involves understanding 
how students employ the use of digital technologies to search for important information about a book and their 
experiences while using the technologies. The rapid advancement in technology and the introduction of digital 
learning platforms have brought about numerous advantages that will be outlined in this review. 


Digital book library 

Digital book libraries have made significant contribution in the development of early reading skills in 
children. Moreover, the digital book libraries enable the students with visual impairments or language-based 
disabilities to easily access reading materials. The ease with which these groups of children are able to read is 
promoted by the use of text-to-speech feature. Synchronized highlighting of texts helps in drawing the attention 
of students to some important words (Biancarosa & Griffiths, 2012). The digital reading tools provide students 
with practice opportunities and individualized feedback; which is critical to improving their reading skills. 


Digital reading program (myON) 

Digital technologies and interactive media have continued to enhance learning in children (Radich, 
2013). Basic applications that are available in digital platforms such as text-to-speech features and the use of 
internet for collaborative learning have contributed in improving the learning experience of many children. 
myON program is one of the key reading interventions that has helped to improve the reading ability of 
elementary school students. MyON facilitates the identification of the Zone of Proximal Development (ZPD) 
and it also suggests the most appropriate books that can be used by elementary school students for independent 
reading. Further, myON helps to enhance reading skills and comprehension, and it motivates students through 
extrinsic rewards (Brekhus, 2011). 


Usability Evaluation of myON 

Usability tests are critical to evaluating the ease with which students are able to access and utilize 
digital libraries to enhance their reading. Prior to using myON, students are required to take the Lexile 
Placement Exam and Interest Inventory. The lexile score obtained helps the program to determine the books that 
are most suited to meet the interest and proficiency of the student (MyON, 2018). 

Another critical feature of myON is that it is capable of providing valuable data regarding the number 
of books that an individual has opened and read, the amount of time spent reading, the quizzes taken and 
progress made in reading. myON has a plan that can be used to enhance student achievement as well as growth 
in reading. The usability tests serve to evaluate the ease or difficulty with which elementary school students are 
able to find a book or information about a book using myON. The usability tests help to assess the experiences 
of students following their interaction with myON website. myON provides quality digital books that contain 
multimedia supports, active reading tools, news articles, as well as real-time assessments for students. 

This project focused on establishing a proper understanding about the usage and compliance of 
students with myON. The myON usability tests were used to evaluate the effectiveness and efficiency of myON 
and the satisfaction obtained by users by using the myON website. The information obtained would be used to 
improve the website. It is critical for the school administration and teachers to devise instructional decisions that 
can be used to promote growth in student learning. 

The usability study was designed to determine interaction of users with regards to three key aspects; 
effectiveness, efficiency, and subjective satisfaction. In order to obtain the usability data, a set of tasks were 
given to the users. Data was then obtained on the ability of users to find a book using the application. 
Participants were also given a post-questionnaire and a SUS survey in order to gather quantitative data on the 
satisfaction of the participants with myON (Brook, 1986). The usability tests helped to identify the concerns of 
the participants and to obtain recommendations that would be used to improve the design of myON in order to 
make it more user friendly. They key questions that were developed and which helped to guide the evaluation 
included: 


1. How effectively do participants complete predefined tasks on myON? 
2. How efficiently do participants complete the predefined tasks on myON? 
3. What is the satisfaction level of the users with the myON? 


4. What is the difference in effectiveness and efficiency rates between the experienced and non- 
experienced myON users? 


Method 


The methodology used for this study was both qualitative and quantitative in nature. A mixed method 
was used so as to help understand more about the effectiveness and efficiency of myON, and the satisfaction of 
participants with myON website. The mixed method would also help in obtaining critical information that 
would be used to improve myON website where necessary. 

A moderated-in-person usability study was carried out so as to evaluate how participants interacted 
with myON website. This was important in understanding the relevance and accuracy of information contained 
in the website. The ten elementary students were each given a set of pre-defined tasks in order to determine the 
effectiveness of the website tools. The interview session with each participant lasted approximately 45-90 
minutes. This time was adequate to gather all the necessary information. The time was also not too much so as 
to create disinterest among the participants. 

Interview method was used because it is widely acknowledged as one of the best methods that can be 
used to obtain first class data that has no interferences (Erickson, 1986). The participants were then asked post- 
test questions with the aim of obtaining qualitative feedback on myON. Finally, the users were taken through a 
SUS survey and a semi-structured interview, which took approximately five minutes each. The two approaches 
were used to collect qualitative data on the satisfaction of participants with myON. 


1.1 Pre-Defined Tasks 


Task 1: I want you to discover the MyON website before we start and tell me about this website. 

Task 2: I want you to login to MyON using your username (..) and password (12...). 

Task 3: You are now on the MyON page, search for “Big Dinosaurs” book to read. 

Task 4: You need to read, “ What if there were No Bees” and rate the book. How would you do this? 

Task 5: You need to read “Space Leftover” book and share your thoughts about the book. How would 
you do this? 

Task 6: Search for a 3rd book to read under the “Because I Like Seasons and Weather’. 

Task 7: whether “Once Upon a Time” book is in the Teacher Recommend for you to read. 

Task 8: You need to open a new book from books recently opened. How would you do this? 

Task 9: You need to know how much time you spent reading. How would you do this? 

Task 10: You need to read “Motion” book and take a quiz. How would you do this? 


Results 


In order to obtain sufficient data and information that would be facilitate the development of a proper 
conclusion, the study involved 10 students (seven males and three female) who consented to participate in the 
usability tests and survey. Five of the participants were familiar with myON, while the other five were novices. 
Four of the participants were in the fourth grade while six were in fifth grade. Six participants were aged 11 or 
12 years while four were aged nine or ten years. 


1.1 Effectiveness — Task Completion Success Rates 

The rate of success in carrying out tasks using a website determines the effectiveness of a website. 
There were 10 participants with each participant involved in 10 tasks. Therefore, there were approximately 10 
tasks, in general, to be completed by the participants. 9 out of the 10 tasks were completed effectively, resulting 
in a 97% task success rate. The 97% completion rate of the tasks indicates that the website was effective in 
enabling the participants to complete their tasks. However, the average time to complete all the tasks for 
participants with experience was 100%, and the average time to complete all the tasks for participants without 
experience was 94%. It can be noted that three participants without experience were not able to complete task 6 
which required them to search for the 3rd book to read under the ‘Because I like Seasons and Weather’. It was 
observed that even some of the participants who had experience on how to use myON found it challenging to 
complete the task at the beginning. All the other tasks except task 6 experienced a 100% completion rate. 


2.1 Efficiency — Task Completion Time 

The efficiency of a website is determined by the amount of time taken to complete a given task. The 
average time to complete all the tasks for participants with experience was 29 minutes and 14 seconds, and the 
average time to complete all the tasks for participants without experience was 37 minutes and 58 seconds. 
Further, it can be observed that the time period between the longest and the shortest task for participants with 


experience was 7 minutes and 45 seconds, and the time period between the longest and the shortest task for 
participants without experience was 7 minutes and 45 seconds. The average time is taken to complete each task 
ranged between 24 seconds to 8 minutes and 09 seconds. In addition, the participant 9 was the fastest in 
completed all the tasks while participant 6 was the slowest, registering a time of 44 minutes and 59 seconds. 
From this, it can be concluded that participant 9 was more efficient in using the website. All the participants 
were completed task 8 below 1-minute and participant 6 was completed the task in 4 minutes and 20 seconds. 
Finally, task 6 was fairly difficult for 3 participants due to the fact they take a long time to complete the task, but 
the rest of the participants were able to complete the task below the average time of 7 minutes and 41 seconds. 


3.1 The difference in experienced and non-experienced users in terms of effectiveness and efficiency\ 


3.1.1 Effectiveness 

The difference in effectiveness between the two participants groups was based on the success rates of 
completing tasks using myON. The experienced users were effective in using myON, with all the participants 
having 100% completion rate. On the other hand, the non-experienced users demonstrated 94% completion rate 
of the tasks. 

Despite the failure of a few of the non-experienced users (3 users) to effectively complete their tasks, 
the high completion rate in the two groups illustrates that the application was effective in enabling the 
participants to complete tasks. 


3.1.2 Efficiency 

The efficiency was determined by the amount of time taken to complete a given task. The difference 
between the two participant groups was observed in the average time taken to complete all the tasks. The 
average time used by the experienced users was 29 minutes and 14 seconds, while the non-experienced users 
took 37 minutes and 58 seconds. 

However, the two groups of participants had a similarity in the time period between the longest and the 
shortest task at 7 minutes and 45 seconds. 


3.1.3. User Satisfaction Survey Results 

For this usability test, we obtained the user satisfaction using the standard SUS survey. The outcome of 
the survey indicated that the mean SUS score for this usability test was 77.3 which is higher than 68. The overall 
SUS score was above average, this score indicates that the majority of the participants were satisfied with the 
website in enabling them to complete their tasks. 


3.1.4 Interview Results 

In order to get more information and data from the participants to test the website. Post-test questions 
were conducted to understand the user's experiences, how they felt about the website and to certain aspects not 
covered in the tasks questions. All the participants agreed that the website was fun, easy and good for kids to 
read the different book online. 6 out of 10 participants found that task number 6 was the most difficult task, 
which 3 of them failure to complete it. However, 4 participants found that task number 3 was the easiest task 
and that because the navigation was clear and easy to use. One of the participants claimed that “I don’t know 
why... when I used the search engine to search for a book to read, it does not come first”, and because of that 
most of participants took a long time tried to found a book. 


Discussion 


The outcome of this study shows that the participants found some of the tasks to be confusing. As a 
result, various participants did not complete their tasks successfully. For example, three of the users were not 
able to complete task six effectively. The participants felt that myON was not well designed. The users were 
required to click on the library navigation tab that directed them to the library page. Thereafter, the participants 
were required to scroll to the end of the page in order to see the ‘Because I like Navigation’ tab which contained 
a drop down menu. It was observed that some of the participants who were experienced with myON found this 
task challenging at the start, but were able to eventually complete it. Given the challenge faced by the 
participants in navigating through the library page, there is need to redesign the search engines in the website 
and substitute it with a drop-down menu. This will allow for all the essential features to be visible at the top of 
the page and thus minimize the problems encountered in navigating through the pages. 

This study provides significant evidence showing that there is need for the redesign and improvement 
of the myON website. One of the evidence can be linked to the fact that it took a long time for the participants to 
locate a book using myON. For instance, it took one of the participants approximately eight minutes longer than 
other participants to complete the task of finding a book. The participant in this case required more assistance on 


how to use the website. It is therefore recommended that the website’s home page should be redesigned in order 
to make it more clear and easy for users to find the navigation tab. Moreover, in order to eliminate the challenge 
faced by users in in scrolling down the page in order to find the scroll down menu, there is need to ensure the 
visibility of all the important features at the top of the page. This will make it easy for the users to navigate 
through the pages. 

Ten of the participants were found to have split-attention that affected the search process as they 
completed their tasks. This occurred when the participants were forced to split their attention between various 
navigations that had similar link information. The key recommendation that can be adopted in order to avoid the 
split-attention effect is the need to change and redesign one of the search engines to ensure that it has a drop- 
down menu. This will make it easy for the students to search and locate all the books contained in the database. 
As such, the students will have no problem finding the books that they need. 

It was also suggested by some of the participants that if the timer could be initiated when the users 
began reading their books, they would be able to know the amount of time spent reading a book. It is therefore 
necessary to include and activate this feature in myON so as to enable the users monitor the time they spend on 
reading a given book. 

This study was able to determine the difference in the effectiveness with which the experienced and the 
non-experienced users were able to complete their tasks using myON. The participants who had prior 
experience with myON were able to complete their tasks successfully. The participants who did not have 
experience with myON were reported to successfully complete an average of 94% of the tasks. Even though 
completing 94% of the tasks is considered to be significantly high, there is need for first time users to gain more 
experience with myON so as to increase their success rates. As outlined earlier, some of the key issues 
associated with the lack of effectiveness among the non-experienced users include design problems, 
mislabelling, and navigation issues. 

The efficiency rates showed that the experienced participants were able to complete their tasks by 
spending less time compared to the non-experienced participants. This served to indicate that the non- 
experienced users require more assistance in understanding the structure of the site and how to efficiently use 
the website. Further, it was observed that the overall SUS score was above average. The above average score 
indicated that more than half of the participants were contented with the website in facilitating their ability to 
complete their tasks. However, the score was not perfect and this illustrates the need for further modification 
and improvement of the website, which has been highlighted in the recommendations. The findings from this 
study also showed that the average learnability was approximately 57.5%. This indicates that the participants 
were not fully knowledgeable on how to use the website. As such, majority of the participants still needed to be 
guided no how to use the website. 


Conclusion 


myON is a form of digital library that has been developed to meet the requirements of the individual 
learners. myON provided all the essential information needed by parents and teachers to monitor and assess the 
progress of students as they read. The usability evaluation contained in the website is critical in determining the 
usability and effectiveness of the services offered by the website. 

This study has evaluated the efficiency, effectiveness, and satisfaction of participants in employing the 
use of myON to complete various tasks. The methods used in the evaluation process include semi-structured 
interviews, observation by researchers, and usability scale satisfaction survey. 

The outcome of the efficiency evaluation showed that the average time used to complete all the tasks 
by the participants with prior experience was approximately 29 minutes and 14 seconds. The participants 
without experience on the other hand spent an average of 37 minutes and 58 seconds to complete their tasks. 
The ability of the users to effectively and successfully complete their tasks was influenced by the split-attention 
effect in the search engine. The split-attention effect occurred when the participants were required to divide their 
attention between the various navigations that had similar link information. 

Further, the effectiveness rate in using myON showed that 100% of the participants with experience 
were able to effectively complete their tasks. Approximately 94% of the participants without experience were 
able to complete their tasks effectively. Three participants without experience found difficulty-completing task 
six that required them to find a book to read under the section ‘Because I like Seasons and Weather’. It was also 
observed that some of the participants with experience on how to use myON faced difficulty in completing tasks 
at the beginning. Apart from task six, all the other tasks reported 100% completion rate. The average SUS score 
was approximately 77.3, which was above the overall average of 68. This illustrated that a large number of the 
participants were satisfied with how the website enabled them to finish their tasks. 

There was a common consensus among all the participants that myON was easy and fun to use, and 
that it was suitable for students who had difficulty in reading books online. There was no significant difference 


in the usability scores between the two groups with regards to effectiveness, efficiency, and satisfaction with 
myON. 

For future research, it is recommended that eye-tracking device should be used to assess the way in 
which users interact with myON. This is due to the fact that the eye-tracking device provides accurate data that 
can aid in identifying where users are looking and the amount of time spent in specific areas on the screen. 
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Abstract 


This study attempted to examine undergraduate students’ problem solving and creative thinking abilities 
when they faced with the task of solving the complex communication problems about the fields of communication 
management, communication design, and communication technologies during the face-to-face learning classroom 
environment. The participants, 89, of the study are the fourth year senior students of the communication sciences 
faculty of Anadolu University, Turkey. The problem solving teams consist of seven students who are willing to 
participate in the study. The participants’ achievements and motivations are the main questions of the research. 
Finally, the study explored the relationships between characteristics of the problem task, and interactions among 
students. 


Introduction 


The problem-based learning (PBL) is a teaching strategy that aims to move the classrooms from an 
instruction-based to problem-based learning environments. The term of PBL has taken a very important role ever 
since constructivist paradigm became so popular at the beginning of 1990s. The educational technologists and/or 
the instructional designers have developed many constructivist problem-based learning environments for formal 
and informal educational settings. When we look at the literature on PBL, we have found the first use of the PBL 
approach in medical sciences. The educators at the medical school of McMaster University in Canada have 
designed a program that has used problem-based instruction method in the 1970s. The PBL strategy and method 
quickly became widespread over to other medical schools. Throughout the 1980’s the faculties introduced the PBL 
methods into many medical and professional schools across in North America, and Europe. In the 1990’s the PBL 
methods spread to the faculty in the natural sciences. The educators in the humanities and social sciences moved 
more slowly in introducing the PBL method over the past decade (Burch, 2000), therefore it could be said that 
there were not many researches on the PBL in the social sciences area during those times. The PBL approach 
requires students to take their own responsibility for their learning as described in the constructivist theory of 
learning. The students are responsible for their learning in collaborative groups by solving concrete real world 
problems. The teacher’s role in this setting is to guide the students’ inquiries by asking Socratic questions. The PBL 
asserts that most students will better learn if they need it; the need arises as they try to solve the problems among 
the competitive teams. 

The literature commend that competition is still a key element which highly motivates students to engage 
in the gamification tasks during the problem finding and problem solving cases among the competitive teams. 
There are still disagreements over the effectiveness of gamification on student learning, but many researchers have 
investigated the effects of gamification for learning and education and found a positive relationship between 
gamification and desired outcomes. Achievements of learning are one of the most important outcomes in learning 
and education (Kim S., Song K., Lockee B., Burton J., 2018). Many researchers have revealed that gamification for 
learning can improve achievement of the learners. Kim and others (2018) investigated that gamification can 
enhance higher order thinking skills, declarative knowledge and procedural knowledge, and test performance in the 
schools. In addition to these research results, some other researchers found the effectiveness of gamification on 
inducing psychological and behavioral changes. Hakulinen et al. (2013), Kumar and Khurana (2012), Li et al. 
(2012), that they all claim that gamified learning environments foster students’ and learners’ motivation and 
engagement. 

In response to these research results, the researcher have decided to integrate game elements into the 
course contents that the students in the competitive teams would be willing to use in their problem solving 
practices. The following research questions guided the current study: 


1) Is a gamification model effective in motivating learners in the competitive groups (teams) to complete 
more problem finding and problem solving activities? 
2) Is a gamification model effective in motivating learners to increase the participants’ motivations? 


3) Is a gamification model effective in motivating learners to increase the participants’ achievements? 
The study and the participants 

Anadolu University’s Department of Communication Design and Management in the Faculty of 
Communication Sciences offers a course named “Creativity and Problem Solving” in the spring semester. This 
(2019) spring semester 82 senior undergraduate students have chosen this course as a core course of the 
department. During the course, each group of students (problem solving teams) has to develop a creative 
solution to the given complex communication problem by the instructor. The instructor chooses these problems 
from the communication field with the help of communication researchers. In addition, the problem solving 
teams have to develop creative complex problems on the given contents. In order to answer the research 
questions, the researcher will use some data collection tools. The first tool is a motivation scale developed by 
Dinger and Doganay in 2016. The scale has 27 items, and 5-point Likert scale to measure the participants’ 
motivations. A group of volunteer communication experts will investigate and assess the competitive teams’ 
problem finding and solving activities to evaluate the findings. Interviews with the students about all sections of 
the course are the other data collection method for the study. 


Findings 


Table 1. Team Working Motivation Factor Average Scores 


TWM 


m2 m3 m4 m6 ml0 ml4 ml7 ml1g ml19 m23 m24 m28 


Mean | 3,63 4,45 3,90 4,28 3,59 3,70 2,67 3,34 3,65 3,83 4,12 3,61 


The first factor of the questionnaire is the motivation factor related to teamwork. The average of the 
substances in this factor is shown in Table 1. When the scores of the questions in table 1 are examined, the 
lowest score in this factor is 2.67. The question with the lowest score in the factor related to motivation is “I 
like to study alone”. As can be seen from this, the participants do not like studying alone. This situation 
expresses us that teamwork can increase the motivation of the participants. 

The highest score in this factor is 4.45, and the question with the highest score in the factor related to 
motivation is “I do care about the ideas of other friends in teamwork”. The results of the question showed that 
the participants cared about the ideas of other friends in teamwork. 

When the teamwork and motivation questions are examined in general, it can be concluded that the 
participants love teamwork and think that doing their other lessons in this way would contribute positively to 
their motivation. 

Table 2. Gamification Motivation Factor Average Scores 
GM ml m5 m9 mll m13 m21 m22 m25 m27 m29 
Mean | 3,68 3,80 3,39 3,44 3,71 3,61 3,38 3,54 3,37 3,78 


The second factor of the questionnaire is the gamification motivation factor. The average of the 
substances in this factor is shown in Table 2. When the scores of the questions in table 2 are examined, the 
lowest score in this factor is 3.37. The question with the lowest score in the factor related to gamification 
motivation is “The content of my gamification design was clear to me”. According to this result, the 
participants do not seem to complete understand the content of the game design. Therefore, in similar situations, 
it is recommended that the design content should be described to the participants very well. 


The highest score in this factor is 3.80, and the question with the highest score in the factor related to 
gamification motivation is “The gamification method described in the course helped me to understand the 
course contents”. The results of the question showed that the gamification method could help participants 
understand the lesson better. 

When the gamification and motivation questions are examined in general, it can be concluded that the 
participants can learn better through gamification, but game design issue needs to be better explained to the 
participants. 


Table 3. Gamification Achievement Factor Average Scores 


The third factor of the questionnaire is the gamification achievement factor. The average of the 
substances in this factor is shown in Table 3. When the scores of the questions in table 3 are examined, the 
lowest score in this factor is 3.50. The question with the lowest score in the factor related to gamification 
achievement is “I can relate the content of gamification design to the issues I face in my own life”. According to 
this result, the participants do not seem to complete understand the content of the game design as the factor 2 
that is related with the design issue. Therefore, in similar situations, it is recommended that the design content 
should be described to the participants very well. 

The highest score in this factor is 4.38, and the question with the highest score in the factor related to 
gamification achievement is “Success in gamification design makes me happy”. The results of the question 
showed that the gamification design could help participants be happy, and motivate them to understand the 
lesson better. 

When the gamification and motivation questions are examined in general, it can be concluded that the 
participants can learn better through gamification, but game design issue needs to be better explained again and 
again to the participants. 


Discussions and Conclusions 


The results of the research show that the teamwork method has a significant effect on student 
motivation in the lessons, especially in the communication related courses. Contrary to popular belief, teamwork 
contributes positively to students' motivation. Students express that learning is more realistic and more fun ina 
gamified learning environments. However, the design of gamified learning environments is the most difficult 
issue for students and even for the researchers. According to Dicheva, Dichev , Agre, and Angelova (2015) 
there are many publications on the use of gamification in education but the majority are only describing some 
game mechanisms and dynamics and re-iterating their possible use in educational context. This means that 
serious research has not yet been conducted on the use and effects of gamification in education. As a result, the 
researcher may say that the students could learn better through gamification, but game design issue needs to be 
better explained repeatedly to the students, researchers, and designers. 
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Abstract 


This paper intends to reveal the results of a study that examines the relationship between the MOOC 
participants demographics and their reasons to take these courses as well as their preferences of the learning 
activities/materials. It is actually a part of a larger study that intended to reach the effective MOOCs design 
principles for Turkish audience. This part of the larger study focused on investigation of the Turkish MOOCs 
participants’ reasons to take these courses and their preferences of the learning activities/materials. Total 754 
learners voluntarily completed the online survey developed to collect data on these reasons and the participants 
demographic characteristics. The results have shown that the learners mostly take these courses to learn more about 
the topics they are interested in. Remarkably, very few participants noted the interaction with others as an 
activity/material they prefer in these courses. 


Introduction 


The Massive Open Online Courses (MOOCs) has been one of the major contemporary hot topics in 
education, particularly higher education since the first offering by George Siemens and Stephen Downes in 2008. 
Although they seem transforming into more self-paced, professional development and degree-oriented courses, all 
around the world there is still a big interest in both supply and demand sides (Shah, 2017). Latest figures reveal that 
currently there are more than 8000 MOOC offerings from more than 800 institutions to around 60 million users 
globally. 

The demand, or motives for registration to MOOCs, has been studied over the last 5 years in various 
settings (e.g.; Agarwal, 2012; Kolowich, 2013; Rice, 2013). Studies have shown that the major motives for MOOC 
participants, or MOOCers are curiosity for exploring the developments in the fields of their own personal interests 
(Kay et al, 2013; Seaton at al, 2014), getting a better insight about what they have been doing in their jobs (Wang & 
Baker, 2015), experience online learning, receive a certificate, and development of their professional knowledge and 
skills (Yuan & Powell, 2013), requiring no prior degree or certificate to participate (Vail, 2013), learning how to 
design online courses as well as learning online pedagogy (Blake, 2013), having a course experience with a small or 
no cost (Chen, Barnett, & Stephens, 2013; Wilson et al., 2013), and taking a course from a prestigious institution, 
famous or well-known person (Adamopulos, 2013). On the other hand, although MOOCs are considered as 
destructive innovation, a big majority of them adapts an instructional strategy: basically, consists of videos and 
readings as conveyors of information, limited -usually unmonitored- peer interaction, and some instructor-student 
interaction, quizzes and exams. Alexander and Boud (2016) claimed that didactic traditional teaching strategies have 
often been implemented in ODL courses and pointed to the potential of online learning being lost as a result. Is it 
true for MOOCs? Do the MOOCs usually employ the same didactic traditional teaching? What are the alternative 
instructional strategies that are more suitable to MOOCs and online courses? Does culture, in general and/or 
learning culture (learning habits and attitudes common in a society) has an effect on designing MOOCs? 
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These are among the questions we want to examine in a larger study focusing on investigating design 
principles for MOOCs specifically offered in Turkish. One part of this research project is about learning the Turkish 
MOOCs participants’ reasons to take these courses, their preferences of the learning activities/materials, and 
relationship between these reasons and the learners’ demographic characteristics. 


Method 


Anadolu University has been offering MOOCs in Turkish to mainly Turkish speaking audiences since 
2015. AKADEMA is the name of the platform created to offer these courses. Currently there are around 80 courses, 
about variety of topics including science, sports, fine arts, writing skills, Turkish folk music, playing musical 
instruments, etc., run three times in a year in AKADEMA. This study was conducted with the participants of 
AKADEMA courses. 

In this descriptive study, an online questionnaire was used to collect data. The related part of the 
questionnaire was developed based-on different previous studies about MOOCs participants’ reasons or motives to 
take these courses, such as Milligan & Littlejohn (2017), Neubéck, Kopp, & Ebner (2015). The related part of the 
questionnaire provided a list of reasons (motives) for participants to indicate their reasons to attend the MOOCs and 
asked to indicate the most relevant top 3 reasons for them. It also included a list of activities/materials presented in 
MOOCs and the participants were asked to indicate their level of preference toward these activities/materials. The 
data collection instrument additionally consisted of questions about the learners’ demographics. The other parts of 
the questionnaire included questions about the participants’ evaluations, study habits, future plans to attend, their 
preferences of the topics for future, and so forth. 

The questionnaire was shared with all the AKADEMA learners and kept open during | April — 30 May 
2018. Total 754 learners voluntarily completed the online questionnaire from all over the country. Mostly 
descriptive statistics were used to analyze the data. Analyzes about the other parts of the study is still undergoing. 


Findings 


The reporting of the findings is organized according to the research variables (demographic characteristics 
and the participation reasons and preferences) and the cross tabulation of the findings. 

Figure | summarizes the age range of the participants. When the table is examined, it is seen that 61.9% of 
the participants are in the 20-39 age range; and 29.8% between in the ages of 40-54. It is interesting to note that the 
percentages of the older (55-73 ages) and the younger (17-19 ages) students are quite low (5.7% older and 2.6% 
youngers). Additionally, as can be seen in the table, there is not a big gap in terms of the gender. Of the 52.3% 
participants were female and 47.7% were male. 


55+ 17-19 
6% 4 2% 


40-54 
30% 


20-39 
62% 
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Male Female 
48% 52% 


Figure I. Demographic information for MOOC participant 


Figure 2 presents the detailed data about the MOOC participants’ top three reasons for taking AKADEMA 
courses. As can be observed, the first three statements that the participants considered as relevant, or important 
reason to take these courses, were ‘learning more about the topics they are interested in’ (90 percent), ‘having a 
valid certificate’ (68.5 percent), and ‘earning a credit that may use in a formal degree program’ (68.5 percent). 


250 
200 
150 
100 
i i 
> So [J all 
learning more about the topic of having a certificate earning credits 
interest 


MNotatall M#Notreally MNotsure Relevant mHighly relevant 
Figure 2. Top 3 reasons to participate AKADEMA MOOCs 


On the other hand, figure 3 shows the participant learners’ preferences of the different types of learning 
activities. As can be drawn from the figure, a big majority of the MOOC participants still prefers readings (87.9 
percent) and videos (87.9 percent). Meanwhile the learners indicated lowest preference to the learner-to-learner 
interaction (49.1 percent). Interestingly, synchronous interaction with the instructors was also preferred less than 
others (71.2 percent). 
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Readings Videos Interactive Assignments Asychnronous  Sychnronous _ Intraction with 
materials interaction with interaction with other students 
an instrcutor an instrcutor 


MNotatall mNotreally MNotsure Relevant mHighly relevant 
Figure 3. Learners’ preferences on the learning materials/activities 
Figure 4 gives some details about the top 3 reasons for MOOC participation and gender. One can infer 
easily infer that there is no significant difference between female and male students’ motives. A similar finding was 
observed in the activity/material preferences of the female and male students. A noteworthy difference between 


genders was seen in the preferences of learners on learner-to-instructor interactions. Figure 5 also shows that male 
students are less willing to interact with the instructors either asynchronously or synchronously. 
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Figure 4. Gender and the top 3 reasons to participate MOOCs 
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Figure 5. Gender and activity/material preferences 


In terms of age groups, a great percent (90.3% relevant and very relevant answers) of those who are 
between in 17-19 ages indicated that the learning more about the topics they are interested in as the top reasons for 
taking AKADEMA MOOCs. Interestingly, all the elderly (55+) students also indicated this reason as the major 
reason for taking these courses. Similar results can be seen for other age groups, too. 


Table 1. Age and the top 3 reasons to participate MOOCs (percentages) 


Reason Age Not at all Not really Notsure Relevant Very 
relevant 
Learning more about the 17-19 6.7 0 0 20 73.3 
topic of interest 
20-39 1.3 1.3 6.6 21.1 69.7 
40-54 1.9 1.9 25 17.2 76.4 
45+ 0 0 0 0 100 
Having a certificate 17-19 13.3 6.7 0 46.7 33.3 
20-39 3.9 5.3 5.3 43.4 42.1 
40-54 6.3 3.1 3.8 34.6 52.2 
45+ 0 0 14.3 0 85.7 
Earning credits 17-19 14.3 0 28.6 42.9 14.3 
20-39 5.4 8.1 17.6 43.2 25.7 
40-54 9.4 5 16.4 39.6 29.6 
45+ 0 0 14.3 14.3 71.4 
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In terms of age groups and activity/material preferences of the students, all of those who are between in 17- 
19 ages preferred using readings and videos in their MOOCs. This group indicated peer interaction (42.8 percent) 
and asynchronous interaction with an instructor (71.5 percent) as the least preferred activities/material for MOOCs. 
Meanwhile, those who are between in 20-29 ages preferred the videos (95.7 percent) over readings (93.4) but still 
reading is the second preferred activity/material among this group. Similar to the previous group, this age group also 
indicated lesser preference of interacting with the other students (41.4 percent). However, it was interesting to notice 
that synchronous interaction with an instructor (67.5 percent) was lesser preferred activity/material than 
asynchronous interaction (76 percent). Additionally, this group also did show a lesser preference for assignments 
(69.7 percent). Very similar findings were observed among 40-54 years old students. They also highly preferred 
readings (100 percent), videos (84.8 percent) and shown less interest in peer interaction (53.1 percent) and 
synchronous learner-to-instructor interaction (75 percent). Finally, the elderly group shown interesting preferences 
although the peer interaction (57.2 percent) still indicated as the least preferred activity/material. This group has 
shown great interest in learning from assignments and feedback provided to their works (100 percent). Interestingly, 
this group also slightly preferred asynchronous interaction with the instructors (76.4 percent) over synchronous 
interaction (75 percent). 


Table 2. Age and activity/material preferences (percentages) 


Activity/Material Age Not at Not really Notsure Relevant Very 
all relevant 
Readings 17-19 0) 0 0 64.3 35.7 
20-39 0 0 6.7 46.7 46.7 
40-54 1.9 3.2 8.2 36.7 50 
45+ 0 0 0 14.3 85.7 
Videos 17-19 0 0 0 42.9 57.1 
20-39 1.3 1.3 11.8 43.4 52.3 
40-54 2.5 3.2 9.5 33.5 51.3 
45+ 14.3 0 0 28.6 57.1 
Interactive materials 17-19 6.7 6.7 0 40 46.7 
20-39 1.3 1.3 10.4 44.2 42.9 
40-54 1.9 3.8 12.7 38.2 43.3 
45+ 0 0 14.3 28.6 57.1 
Assignments 17-19 0 0 7.7 46.2 46.2 
20-39 3.9 2.6 23.7 36.8 32.9 
40-54 3.2 5.1 11.5 38.9 41.4 
45+ 0 0) 0 28.6 71.4 
Asynchronous interaction 17-19 7.1 7.1 14.3 42.9 28.6 
20-39 1.3 6.7 16 42.7 33.3 
40-54 4.5 5.7 13.4 41.4 35 
45+ 0 0 14.3 14.3 71.4 
Synchronous interaction 17-19 0 7.1 14.3 42.9 37.5 
20-39 1.3 9.2 22.4 27.6 39.5 
40-54 5 44 15.6 30..6 44.4 
45+ 0 5.3 14.3 14.3 71.4 
Peer Interaction 17-19 0 35.7 21.4 21.4 21.4 
20-39 5.3 21.3 32 26.7 14.7 
40-54 8.2 17.1 21.5 32.3 20.9 
45+ 14.3 14.3 14.3 14.3 42.9 
Conclusions 


MOOCs nowadays stand out as widespread and popular digital education applications on a universal 
scale. Innovative practices created by new media technologies force traditional education paradigms to change. In 
this context, with the development of technology, traditional education patterns are broken, and MOOCs practices 
appear as an alternative educational environment. MOOC s are becoming more widespread in order to make more 
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information available to the mass circulation and to make the courses in universities accessible and flexible for 

the wider audience. It would be wrong to indicate that examining the MOOCs phenome might help the governments 
and related institutions develop policies for effective and efficient integration of information and communication 
technologies into educational systems. 

The goal of this study was to determine the learners’ reasons for taking the Anadolu University’s MOOC 
offerings in AKADEMA platform and their preferences of learning activities/materials. The top 3 motivation of the 
participants to take AKADEMA MOOCs was identified respectively as learning more about the topics of interest, 
having a valid certificate from a well-known institution, and earning credits that can be used in a degree program. 
This result supported the previous studies conducted in Turkey and all around the world. For instance, Aydemir and 
Celik (2018) conducted a study to determine the reasons for joining the MOOCs and the participants of that study 
also stated that they took those course because they were interested in learning new things, developing themselves 
and being interested in the subjects of the courses they took. In another study, Aybek (2016) the same phenomenon 
with the 24-40 age group and found out that the learners attended the courses owing to their interest and curiosity 
towards the courses. Similar findings were also indicated the global MOOCs literature. Learning more or gaining 
more experience and knowledge on the topics of interest was identified as one of major motives for taking MOOCs 
in many studies, such as Agarwal (2012), Allon (2012), Belanger and Thornton (2013), Breslow et al. (2013), Evans 
(2012), Fini (2009), Kaul (2012), Kolowich (2013), Rice (2013), etc. Meanwhile, the literature clearly presents that 
learners’ performance in MOOCs is highly correlated with the learners’ expectations in joining the courses (Hew & 
Cheung, 2014; Oktal, 2013; Venkatesh & Davis 2000). This might be considered as a rational for the finding about 
the learners’ participation to MOOCs because of learning more on the topics of interest. Additionally, theory of 
adult learners also indicated that adults’ one of the biggest motivations towards learning is learning on their topics of 
interest. 

In terms of learning activities/materials, it was very surprising to see readings as highly preferred 
activities/materials as videos even among younger students. This result might be associated with the traditional 
distance education background of Anadolu University. Anadolu University distance education programs and courses 
mainly require self-paced study and asks students learn from specially designed, written, and published textbooks 
although nowadays more digital media are being presented. So, the learners learn by reading their textbooks and 
take centralized exams in order to pass their courses. This design could be effective on the MOOCs participants’ 
preferences of the learning activities/materials in this study since a good deal of learners in MOOCs is also distance 
education students. 

The analyses about effects of the gender and age of the participants on their motives and preferences of 
activities/materials have shown no significant results. In other words, there is no significant difference between 
female and male students’ motives. A similar finding was also observed in the activity/material preferences of the 
female and male students. A notable difference between genders was seen in the preferences of learners on learner- 
to-instructor interactions. The male students are less willing to interact with the instructors either asynchronously or 
synchronously than females. 

In the light of the results, it was concluded that readings and videos are interestingly important 
activities/materials for all age and gender groups. Although MOOCs promote technology-based learning, learners 
continue their traditional distance learning behaviors. Even in technology-rich, technology-based learning 
environments, the Turkish students download the materials and study mostly printed versions of those materials. So, 
it can be claimed that the Turkish MOOC participants carry their traditional learning behaviors into MOOCs. 

A remarkable result of this study is about the participants’ preferences regarding the learner-to-learner 
interaction in MOOCs. It is the least preferred learning activity/material among the participants of this study. 
Actually, Aydemir and Celik (2018) also found a similar result in their study. One can easily relate this result with 
again the previous distance learning experiences of the learners. Anadolu University as well as many other distance 
education providers do not focus on learner-to-learner interaction rather prefer learner-to-content interaction. So, 
again, the learners carry their learning habits into MOOCs in this case and prefer less peer interaction. It would be 
interesting to create and offer a cMOOCs to Turkish community and see the effectiveness and engagement. This 
result might be considered as a base while designing a MOOC for Turkish learners. Also, it is a good topic for 
further research on MOOCs for Turkish learners. 

Another interesting result is about the earning a valid certificate from a well-known institution. Those 
participants who are between in 20 and 39 ages indicated this as the most relevant motive for taking AKADEMA 
MOOCs. This result can be correlated with the high unemployment rate among new graduates of higher education, 
and the raise of the informal and non-formal learning and their recognition among employers. In other words, along 
with a university degree diploma, certificates from well-known institutions are becoming added values for getting a 
job. This might be an interesting clue for MOOCs designers and providers same as above conclusion: 
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Abstract 


The purpose of this study was to determine the effects of technology-supported tools used in 
educational environments to support cognitive learning processes on educational outcomes in design-based 
learning environments. In this context, Academic Search Complete, Eric, Ulakbim, Google Scholar and CoHE 
search engines were scanned. Scans performed; Digital story, Game design (Kodu), Algorithm and 
programming (Scratch), 3D design (Tinkercad, 3D print), Mobile programming (AppInventor) and Embedded 
programming (microcontroller) arguments. As a result of the scanning, 164 studies were obtained. The papers 
accessed by researchers were examined by taking into consideration the inclusion and exclusion criteria. In this 
paper, motivation, attitude, achievement, class participation, metacognition and 21st century learning of design- 
based learning environments were examined in terms of dependent variables in studies used design based 
learning environments as a primary implementation context. 


1. Introduction 


Design-based learning (DBL) environments are environments that allow the individual to create their 
own cognitive processes according to their learning preferences, styles and skills (Doppelt, Mehalik, Schunn, 
Silk, & Krysinski, 2008). The most important feature of these environments is that the individual creates his / 
her own experiences with the active participation of the process. These environments are based on constructivist 
learning theory as it is based on the active participation of the individual. In the design of these environments, 
the constructivist learning theory offered; problem/project, related cases, information resources, cognitive tools, 
conversation/collaboration tools and social/contextual support steps to be implemented. 

DBL also supports the transfer of cognitive information since it is based on learning through 
experience and a product is created at the end of the process. In order to ensure the transfer of cognitive 
knowledge to complex, new and real situations, students should be supported with cognitive tools (Jonassen, 
1999). Cognitive tools can be in the form of hands-on activities or technology-supported. Use of technology- 
supported cognitive tools in learning environments are effective for development of academic achievement, 
critical thinking, problem solving, information literacy and collaboration skills (Jonassen & Reeves, 1996; Kim 
& Reeves, 2007). DBL environments have a positive effect on many learning outcomes due to active 
participation of learners in the design and development process, the development of a product, and the support 
of technology-supported cognitive tools. In this context, it is aimed to determine the effect of DBL 
environments by using different technology supported cognitive tools on educational outcomes. 


1.1. The Importance of Usage Design-Based Learning in Education 

Design-based learning (DBL) provides many different advantages in learning environments. Since 
DBL provides a relationship with the real life of the learners in structuring their knowledge, it enables the 
learners to be motivated against learning (Doppelt, 2003). The learners actively participates in these 
environments. When the learner actively participates in the learning process, the student makes sense of the new 
information by matching his / her knowledge with his / her existing experiences (Driscoll, 2005b). The DBL 
environments aim to learn with the experience advocated by constructivist learning theory. These experiences 
give students not only cognitive skills but also metacognitive skills such as self-esteem and personal 
responsibility (Waks, 1995). 

DBL environments provide a learning environment in which instructor and learners have different roles 
traditional lecture based teaching. In lecture based teaching environments, the instructor has the role of actively 
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giving information, and the learner has the role of passively receiving information. On the other hand, the DBL 
environments support a student's active participation in the learning process by providing opportunities to create 
products that ensure the development of cognitive strategies and processes. The design process of constructivist 
learning environments proposed by Jonassen (1999), as in constructivist learning theory, should be taken into 
consideration in DBL environments. This design process includes problem /project, related cases, information 
resources, cognitive tools, conversation/collaboration tools, social/contextual support. 

In DBLs, learning process is supported by real life problems or project-based activities that provide 
experiences with the active participation of learners, facilitating and supervising the process of the teacher 
(Gomez Puente, van Eijck, &Jochems, 2013). In order to provide learning with experience in these 
environments, it is also necessary to transfer cognitive knowledge. In order to ensure the transfer of cognitive 
knowledge to complex, new and real situations, it is necessary to provide students with cognitive tools 
(Jonassen, 1999). Cognitive tools can be in the form of hands-on activities in these environments as well as 
technology-supported. Technology-supported learning environments have many contributions to the learning 
process. 

The use of technology in teaching facilitates learning because it supports different types of learning in 
storing and remembering information in cognitive processes (Mayer & Moreno, 1998; Paivio, 1991). In 
addition, the presentation of technology-supported cognitive tools provides support for learning complex 
information and reduces cognitive load (Driscoll, 2005a ). Use of technology-supported cognitive tools in 
educational settings; academic achievement is effective in the development of mental skills such as critical 
thinking, problem solving, information literacy and collaboration (Jonassen & Reeves, 1996; Kim & Reeves, 
2007). 


2.Research questions 


The aim of this study is to determine the effect of DBL environments by using different technology 
supported cognitive tools on educational outcomes. Within the scope of this study, the research questions are as 
follows: 

1. What are the effects of design-based learning environments on students’; 
motivation in different learning topics? 

students' attitude in different learning topics? 

achievement in different learning topics? 

engagement in different learning topics? 

metacognition in different learning topics? 

21st century skills in different learning topics? 


mongoose 


3. Research method 


This study was conducted as a literature review. For the review, Academic Search Complete, ERIC, 
Ulakbim, CoHE (Council of Higher Education) Thesis Center and Google scholar was used for searching 
articles. 


3.1. Search terms and domains 
In order to determine the effect of design-based learning on educational outcomes, searches were made 


in the databases using the topics and keywords in Table 1. 


Table 1. Databases, research topics and keywords searched in literature. 


Databases Topics Keywords used 
Digital story Digital storytelling 
Digital storytelling in education 
Game design (Kodu) Game design with kodu 
Academic Search Complete Kodu in education 
ERIC Algorithm and Programming with Scratch in education 
Ulakbim programming (Scratch) Scratch use in education 
CoHE(Council of higher Algorithm training with scratch 
education) Thesis Center 3D design (tinkercad, 3D Tinkercad use in education 
Google scholar print) 3D print in education 
Sketchup use in education 
Mobile programming Mobile programming with AppInventor 
(AppInventor) Mobile programming with AppInventor 


in education 
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Microcontroller in education 
Embedded programming with micro 
controller 


Embedded programming 
(micro controller) 


3.2. Research selection criteria 

As a result of the literature review, 164 studies have been reached in total in experimental and quasi- 
experimental studies written in Turkish and English languages, except for articles that are not allowed by the 
author and paid in other databases. If there are published articles of the studies carried out as master's or doctoral 
dissertations, this study has also been reviewed. These studies were examined by considering the inclusion and 
exclusion criterias in Table 2. Inclusion and exclusion criterias were determined by the researchers considering 
the characteristics of the DBL environments. 


Table 2. Inclusion and exclusion criterias 


Inclusion criterias 


Exclusion criterias 


Real world problem-based studies 
Project based studies 
Active participation of students 


Experimental and quasi-experimental studies 


Applications for all age groups 

At the end of the teaching, the student put forward a 
product 

Quantitative dimension of mixed research 


Qualitative studies. 

Opinion studies. 

Instructional studies with media developed by the 
instructor. 

Educational estimates of the applications prepared 
with tools. 

Instructional design studies. 

Literature reviews. 


Development studies. 


Formative research. 
Experience studies. 
Presentation studies. 
Study recommendations. 


As a result of the investigations, it was seen that 39 out of 164 studies examined the effects on 
educational outcomes in DBL environments. The design-based researches examined in detail; 12 of them are in 
digital story, 2 of them are in game design with code, 13 of them are in algorithm and programming with 
Scratch, 3 of them are in 3D design with tinkercad or 3D print, 5 of them are in mobile programming with 
AppInventor, 4 of them are in embedded programming with micro controllers. 


3.3. Design based learning researches examined as a result of selection criteria 

The studies on the effect of DBL environments on educational outcomes were examined under the 
headings of research type, research aim, research method, independent variable, dependent variable, data 
collection tools, participants and results. The design-based research studies in the literature were further 
investigated based on their results to answer current study’s research questions. 


4. Results and Discussion 


The use of DBL environments in education has an impact on many educational outcomes. Design- 
based learning enables learners' motivation, attitudes towards the course / material, academic achievement, 
active participation, metacognitive skills and 21st century skills. 


4.1. Research question 1: What is the effect of design-based learning environments on students' 
motivation in different learning topics? 

Motivation is one of the factors that should be considered in the design of instruction. In other words, 
the designer should design the instruction by taking the necessary measures to increase the motivation of the 
learners. Keeping the motivation of learners high is critical for a good instructional design (Martin, 1999). 
Different models are used in the design of instruction to take into account motivation. One of the most well- 
known models is the ARCS model. This model was proposed by Keller (1979). The model consists of four steps 
that should be considered in instructional design to ensure motivation (Keller, 1983, 1987). These steps; 
attention, relevance, confidence and satisfaction. The problem-based trainings, which are designed with ARCS 
model in DBL environments, contribute positively to students' motivation (Koshino et al., 2013). 

Active participation of an individual in the process is highly effective in increasing an individual's 
motivation against the learning environment (McCombs & Whisler, 1997). As DBL environments are the 
environments where the learner actively participates, the learner motivation increases. The increase in 
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motivation results in the researches is due to the active participation of the students by the researchers (Topalli 
& Cagiltay, 2018). The effect of motivation, which is one of the learning outcomes, on different media in the 
DBL environments was examined. It is seen that these tools increase motivation and increase success in DBL 
environments (Chun-Ming et al., 2012; Demirer, 2013; Erol & Kurt, 2017; Topalli & Cagiltay, 2018). There are 
also studies showing the opposite of this situation. Gdgen (2014) concluded in her study that the achievement of 
the students increased but there was no change in their motivation. In this study, where students’ achievements 
increased and they developed positive attitudes, it was thought that students did not give sincere answers to 
motivation data collection tools. Increasing motivation positively develops positive attitudes in DBL 
environments (Demirer, 2013; Gécen, 2014), metacognition skills such as time management, self-test (Gé¢en, 
2014) and also contributes to the development problem-solving skills from 21st century skills (Chun-Ming et 
al., 2012). 


4.2. Research question 2: What is the effect of design-based learning environments on students’ attitude 
in different learning topics? 

Attitude is the cognitive and emotional preparation that has the power of influencing and directing the 
behaviors towards all objects, events and situations of interest that are formed as a result of an individual's life 
and experiences (Alport, 1935). Attitudes in learning environments are influenced by individual's life, 
knowledge level, interests, reinforcers, imitations and social learning. The presence of attitude can be explained 
as a result of the measurement and observation of cognitive thinking process, emotions and behaviors (Inceoglu, 
1993). An individual's positive attitude towards the learning environment will facilitate learning, motivation, 
and acquiring metacognitive or 21st century skills in order to increase student success. 

DBL environments enable students to develop positive attitudes towards technology-based cognitive 
tools, learning topics, and product discovery processes. When the researches were examined, it was seen that 
students developed positive attitudes towards the cognitive tools of AppInventor (Dabney et al., 2013), digital 
story (photo story 3) (Heo, 2009) and Scratch (Chiang & Qin, 2018). In addition to cognitive tools, students 
show positive attitudes towards the subjects they want to learn by actively participating in the learning process 
(Ke, 2014; Wang et al., 2017). DBL environments also enable students to demonstrate positive attitudes towards 
writing skills by creating scenarios as a result of the use of digital storytelling tools (Baki, 2015). It is seen that 
the studies on the attitude of DBL environments on attitude are incomplete in terms of Kodu game design, 3D 
design and microcontrollers and embedded programming. Experimental researches that will examine the effect 
of these areas on attitudes in DBL environments will guide educators for future uses. 


4.3. Research question 3: What is the effect of design-based learning environments on students’ 
achievement in different learning topics? 

Success is one of the most frequently used dependent variables, which are used in most scientific 
researches and the effects of different teaching methods are examined. Success refers to the degree of ownership 
of the knowledge, skills and abilities that are desired to be acquired. Achievement results are one of the main 
variables that indicate whether a designed instruction is effective or not. The effectiveness of a teaching results 
from the level of achievement of the desired learning objectives. While preparing learning objectives, it is 
considered which theory will be developed based on which theory. These theories argue that success is affected 
by many different factors in learning environments. In learning environments; behavioral learning theory argues 
that success is influenced by environmental factors (Driscoll, 2005c), cognitive learning theory is influenced by 
factors in the process of information processing (Driscoll, 2005a ), and constructivist learning theory is 
influenced by the individual's development, experiences and sociocultural structure (Jonassen, 1999). 

Success depends on the performance of the learner. The lower the difference between the existing 
situation and the desired situation in human performances, the higher the success (Van Tiem, Moseley, & 
Dessing's, 2000). Human performance; organization system (instructional design), incentives (reward, 
reinforcement etc.), cognitive support (modeling, coaching, building scaffolding), tools (cognitive tools, 
technology supported tools), physical environment (classroom environment), lack of knowledge / skills, natural 
/ hereditary (intelligence, physical features) capabilities are affected by internal and external factors (Wile, 
1996). 

It is essential for the individual to gain experience in DBL environments. Therefore, in these 
environments, active learning of the learner is ensured and practical learning is provided. Success levels vary 
depending on the learner's active participation. Students' achievement can be increased by using different 
technology supported cognitive tools in DBL environments. In the studies, it is seen that Scratch (Su et al., 
2014) and 3D printing (Dahle & Rasel, 2016), one of the technology-supported learning tools used in DBL 
environments, increase student achievement compared to the environments where teaching strategy is used with 
traditional presentation. In addition to these studies, there are studies comparing the effect of different 
technology supported cognitive tools on success. Omar (2018) examined the impact of microcontrollers, Scratch 
tools and programming success in traditional environments. As a result of the study, it was observed that 
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students using microcontrollers were more successful than Scratch users and the traditional learning group was 
the lowest. Korkmaz (2018) also compared Scratch and Lego Mindstorms Ev3 applications and concluded that 
Lego Mindstorms Ev3 contributes more to success. In addition, AppInventor (Papadakis et al., 2016) and digital 
stories (Bitytikcengiz, 2017; Chun-Ming et al., 2012; Demirer, 2013; Gégen, 2014; Sancar-Tokmak & Incikabi, 
2013) also has a positive effect on success. Students' positive attitude towards the course or application (Cetin, 
2016; Chiang & Qin, 2018; Korkmaz, 2016; Lewis, 2011) and high levels of satisfaction (Chen et al., 2016) are 
among the factors affecting success. Collaboration in DBL environments also increases the success of 
individuals because it enables peer learning (Lewis, 2011). 


4.4, Research question 4: What is the effect of design-based learning environments on students’ 
engagement in different learning topics? 

Participation is the inclusion of the student in the learning processes. In a learning environment, if the 
course is considered important by the student and falls within the student's interest, the student makes an effort 
to make more effort, ie to participate more (De Volder & Lens, 1982; Miller, Greene, Montalvo, Ravindran, & 
Nichols, 1996; Wigfield & Eccles, 2000). Therefore, providing motivation in learning environments is an 
important factor for participation (Skinner & Belmont, 1993). In learning environments, many technology- 
supported cognitive tools can be used in order to attract students' interest and to ensure their participation. 
Considering the students' tendency to technology, it is possible to create the necessary interest for participation 
with technological tools. DBL environments provide learning environments where students' participation is 
actively provided. At the same time, the use of technology-supported cognitive tools in students' activities / 
product discovery processes is also provided by the DBL environments. 

As a result of the research studies conducted on the effect of DBL environments on educational 
outcomes, it was concluded that active participation is seen as the main element in all researches. Providing 
active participation in these environments plays an active role in the development of students' achievement (Su 
et al., 2014; Topalli & Cagiltay, 2018), motivation (Chang et al., 2017; Chun-Ming et al., 2012), positive 
attitude (Dabney et al., 2013; Wang et al., 2017), and development of metacognitive skills (Dere, 2017; 
Saritepeci, 2017) and 21st century skills (Akcaoglu & Koehler, 2014; Pellas & Vosinakis, 2018; Roscoe et al., 
2014). In the researches, the effects of active participation among the effects of DBL environments on 
educational outcomes were not examined. The reason for this is thought to be due to the active participation of 
one of the main characteristics of the DBL environments. Participation in DBL environments positively affects 
the achievement of students of different learning levels (primary, secondary, high school and higher education). 
The use of different technology supported cognitive tools in these environments also plays a major role in 
ensuring participation. It was concluded that digital story creation tools (storyboardthat, powtoon, photo story), 
Scratch, AppInventor, Kodu, Microcontrollers, 3D printing and Tinkercad tools that were examined in the scope 
of this study had an effect on students' active participation and different learning outcomes. 


4.5. Research question 5: What is the effect of design-based learning environments on students' 
metacognition in different learning topics? 

Metacognition is defined as thinking of thinking. Metacognition in Turkish literature; executive 
cognition, metacognition, cognitive awareness. Metacognition is the awareness and control of an individual's 
own knowing processes (Huitt, 1997). Metacognition in learning environments is important in terms of 
contributing to the individual in terms of language development, self-control, writing, memory and problem 
solving skills, knowing which areas an individual is missing, and what his or her own experiences are in order to 
ensure permanence in learning (Flavell, 1979). In order to have metacognition, metacognitive knowledge and 
metacognitive skills are required. In order to have metacognition information, one has to know what he / she 
believes in, the state of the current knowledge and what kind of cognitive activities will be operated (Flavell, 
1979). Metacognition skills are planning, observation, testing, correction, selection and evaluation of specific 
strategies (Brown, Armbruster, & Baker, 1986). 

DBL environments enable students to operate their learning processes by being informed about their 
own learning with active participation. In these environments, students are able to organize their time 
management skills and cognitive knowledge as they seek solutions to problems in a limited time (Gé¢en, 2014). 
Furthermore, Gécen (2014) also states that in addition to time management, students contribute to metacognitive 
skills in terms of processing information, selecting main ideas, identifying study assistants, and self-testing and 
developing test strategies. Self-assessment and self-efficacy knowledge, which is one of the metacognition 
skills, is supported by DBL environments (Baki, 2015; Heo, 2009; Korkmaz, 2016; Liu et al., 2013). In the 
teaching of programming in DBL environments, the student develops his / her planning and organizing skills by 
providing information to the students about how they can structure information and develop information 
processing thinking skills. In addition, the product development process allows the individual to test, correct, 
select and evaluate specific strategies. 
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4.6. Research question 6: What is the effect of design-based learning environments on students’ 21st 
century skills in different learning topics? 

In line with the developing technology and changing learning needs, the skills required for the learners 
today are called 21st century skills. Individuals must have 21st century skills to become well-educated citizens 
(Wangenheim, Alves, Rodrigues, & Hauck, 2017). 21st century skills include the skills, education, attitudes and 
mental habits that today's students must have in order to find work in the future. In the study conducted by 
Sarter (2010) using PISA data, it was concluded that the scores obtained from the exams conducted by OSYM 
in Turkey do not have any relation with academic achievement. In other words, although students are 
academically successful, they fail the exams. Saner (2010) thinks that this situation stems from the fact that the 
educational environments in Turkey are teacher-centered and the student is a passive listener. In line with this 
idea, it can be said that the students have memorized the information about their courses and forgot after some 
time after internalization and transfer. 21st century skills need to be developed to prevent such situations, to 
make students aware of their own learning, and to ensure transfer and persistence. 21st century skills are 
grouped under three main headings: learning and innovation skills, life and career skills, and information, media 
and technology skills, as shown in Figure | (Trilling & Fadel, 2009). The 11 skills collected under these 
headings are called the 21st century skills. 


21st 
century 
skills 


information, media 
and technology skills 


Life and career skills 


Learning and 
innovation skills 


Figure 1. 2\st century skills 


21st century skills can be learned through problems and questions in learning environments (Trilling & 
Fadel, 2009). Small groups cooperative learning method, project-based learning method, problem-based 
learning method and DBL methods can be used in developing 21st century skills in learning environments 
(Bellanca, 2010; Rotherham & Willingham, 2009; Trilling & Fadel, 2009). As DBL environments provide 
project / problem-based learning and students are guided by the teacher during the learning process, they are one 
of the ideal learning environments in teaching these skills. These environments enable students to acquire 
learning and innovation skills, life and career skills, and information, media and technology skills. 

Critical thinking, problem solving, communication and creativity skills of learning and innovation 
skills in DBL environments were discussed in the studies examined. Critical thinking is the ability to make 
explanatory and evaluative information judgments in order to decide how to behave, what to believe, and to 
express these information judgments verbally (Evancho, 2000). Students can make information judgments and 
express them with their practices in DBL environments. In this way, students’ critical thinking skills are 
developed (Chung, 2007; Emert, 2013; Pellas & Vosinakis, 2018; Wang et al., 2017). Problem solving skills can 
be defined as the ability of the student to produce solutions to new problems encountered by using existing 
knowledge. Researches examining the effect of DBL environments on educational outcomes, it is seen that 
problem solving skills can be increased by means of cognitive technology supported tools (Akcaoglu, 2014; 
Akcaoglu & Koehler, 2014; Chun-Ming et al., 2012; Korkmaz, 2018; Kwon et al., 2012; Pellas & Vosinakis, 
2018; Vatansever & Goktalay, 2018).Communication skill is the characteristic that an individual should have in 
order to express his / her thoughts clearly and to take part in group works. It is ensured that students’ 
communication skills are improved through collaborative studies in DBL environments (Moreillon & Hall, 
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2014). The ability to creative thinking can be defined as the use of existing or new information to bring different 
perspectives, innovations and unconventional thoughts to different situations. (Bentley & Yildirim, 2004). In 
DBL environments, students learn from their own experience, which is a requirement of constructivist learning 
theories. As each student's life will vary, their learning will also differ. In this learning, it is influenced by the 
creative ability of the students to propose solutions to the problems faced by individuals by using different tools, 
different scenarios, different products and different algorithms. In these environments, students develop creative 
thinking skills while developing products with problem-based technology-supported tools (Emert, 2013; Pellas 
& Vosinakis, 2018). 

In the experimental and quasi-experimental studies on design-based learning environments, there is no 
direct research that measures life and career skills. However, studies also asserted that students can develop self- 
management and entrepreneurship skills through developing games (Kafai & Burke, 2015), and increase their 
social skills by interacting teachers and peers through collaboration activities. Also; Lewis (2011), in his 
research, argued that productivity could be higher than students' single work. In addition, students gain 
leadership skills in group work. In all of the researches, students have the responsibility skills since they develop 
a solution for a problem / have their own projects. 

The DBL environments also contribute positively to the 21st century skills of information literacy, 
media literacy and information and communication technologies literacy skills. These environments contribute 
to information literacy through the development of reflective thinking and writing skills (Baki, 2015; Ciral1, 
2014). Computational thinking skills can be defined as system design with the use of computer sciences, 
problem solving and revealing human behaviors (Wing, 2006). Students' computational thinking skills can be 
improved with the use of computer-aided tools in DBL environments (Biiyiikcengiz, 2017; Morelli et al., 2011; 
Roscoe et al., 2014). 


5. Conclusions 


As a result of the current study, it was concluded that DBL environments have a positive impact on 
many learning outcomes. However, it was observed that some learning outcomes were not included in the 
examined studies. In the literature, there is no experimental research about flexibility and adaptability, 
assertiveness and self-management, social and intercultural interaction, productivity and accountability, 
leadership and responsibility skills. Similarly, DBL environments have been interpreted as enabling 
metacognitive skills of practice, feedback and evaluation, but experimental studies are not included. In future 
studies, the effects of different technology supported cognitive tools not included in the research can be 
examined. In addition, the number of experimental and quasi-experimental studies in which different 
independent variables were examined was found to be low. In the same way, the number of samples was 
observed to be low. Increasing the number of studies and sample numbers will increase the effect size. In the 
studies, it was observed that evaluations were made frequently with questionnaires, success tests and scales. In 
the literature, very few findings have been encountered for the evaluation of the products developed by the 
students. Since these products are thought to have a major impact on learning outcomes, they should not be 
ignored in future studies. 


6. References 


Akcaoglu, M. (2014). Learning problem-solving through making games at the game design and learning 
summer program. Educational Technology Research and Development, 62(5), 583-600. 

Akcaoglu, M., & Koehler, M. J. (2014). Cognitive outcomes from the Game-Design and Learning (GDL) after- 
school program. Computers & Education, 75, 72-81. 

Alport, G. (1935). Attitudes//Murchison C. Handbook of social psychology. Worcester. 

Altok, T. (2009). Calisanlarin motivasyonunu etkileyen faktorlere iligskin hizmet ve imalat isletmelerinde 
karsilastirmal bir arastirma. Sosyal Bilimler, 

Baki, Y. (2015). Dijital 6ykiilerin altinci simf 6grencilerinin yazma stirecine etkisi. Doktora Tezi, Erzurum: 
Atatiirk Universitesi. 

Barron, B. (2000). Achieving coordination in collaborative problem-solving groups. The journal of the learning 
sciences, 9(4), 403-436. 

Bellanca, J. A. (2010). 2/st century skills: Rethinking how students learn: Solution Tree Press. 

Bentley, T. J., & Yildirim, O. (2004). Takiminizin yeteneklerini gelistirmede yaraticilik (creativity): Hayat. 

Brown, A. L., Armbruster, B. B., & Baker, L. (1986). The role of metacognition in reading and studying. 
Reading comprehension: From research to practice, 49-75. 

Bruner, J. S. (1966). Toward a theory of instruction (Vol. 59): Harvard University Press. 


25 


Buyikcengiz, M. (2017). Dijital dykiileme metodunun ortaokul 6grencilerinin fen bilimleri dersi akademik 
basart, bilimsel siireg becerileri ve derse yonelik tutumlarina etkisi. (Yiiksek lisans), Akdeniz 
Universitesi, Antalya. 

Cetin, I. (2016). Preservice Teachers’ Introduction to Computing: Exploring Utilization of Scratch. Journal of 
Educational Computing Research, 54(7), 997-1021. 

Chang, C.-K., Yang, Y.-F., & Tsai, Y.-T. (2017). Exploring the engagement effects of visual programming 
language for data structure courses. Education for Information(Preprint), 1-14. 

Chen, H.-Y., Nieh, H.-M., Yang, M.-F., Chou, Y.-K., Chung, J.-H., & Liou, J.-W. (2016). Implementation of a 
Low-Cost Automated LED Photometer for Enzymatic Reaction Detection to Teach Basic 
Bioelectronics Technologies in Vocational High Schools. JEEE Transactions on Education, 59(3), 194- 
201. 

Chiang, F.-k., & Qin, L. (2018). A Pilot study to assess the impacts of game-based construction learning, using 
scratch, on students’ multi-step equation-solving performance. Interactive Learning Environments, |- 
12. 

Chun-Ming, H., Hwang, G.-J., & Huang, I. (2012). A project-based digital storytelling approach for improving 
students' learning motivation, problem-solving competence and learning achievement. Journal of 
Educational Technology & Society, 15(4), 368. 

Chung, S. K. (2007). Art education technology: Digital storytelling. In: Taylor & Francis. 

Cirah, H. (2014). Dijital Hikaye Anlatimimin Gérsel Bellek ve Yazma Becerisi Uzerine Etkisi. 

Dabney, M. H., Dean, B. C., & Rogers, T. (2013). No sensor left behind: enriching computing education with 
mobile devices. Paper presented at the Proceeding of the 44th ACM technical symposium on Computer 
science education. 

Dahle, R., & Rasel, R. (2016). 3-D Printing as an Effective Educational Tool for MEMS Design and 
Fabrication. EEE Transactions on Education, 59(3), 210-215. 

De Volder, M. L., & Lens, W. (1982). Academic achievement and future time perspective as a cognitive— 
motivational concept. Journal of Personality and Social Psychology, 42(3), 566. 

Dede, Y., & Argiin, Z. (2004). Ogrencilerin matematige yénelik igsel ve dissal motivasyonlarmin belirlenmesi. 
Egitim ve Bilim, 29(134). 

Demirer, V. (2013). Ilkégretimde e-dykiileme kullanimi ve etkileri. Yayinlanmamis Doktora Tezi, Necmettin 
Erbakan Universitesi. 

Dere, E. (2017). Web Tabanh 3B Tasarim Uygulamalarmin Ortaokul Oérencilerinin Uzamsal Gérsellestirme ve 
Zihinsel Déndiirme Becerilerine Etkisi. Yiiksek Lisans Tezi, Baskent Universitesi Egitim Bilimleri 
Enstitiisti. Ankara. 

Doppelt, Y. (2003). Implementation and assessment of project-based learning in a flexible environment. 
International Journal of Technology and Design Education, 13(3), 255-272. 

Doppelt, Y., Mehalik, M. M., Schunn, C. D., Silk, E., & Krysinski, D. (2008). Engagement and achievements: A 
case study of design-based learning in a science context. Journal of technology education, 19(2), 22- 
39. 

Driscoll. (2005a). Cognitive information processing. In Psychology of learning for instruction (Vol. 3, pp. 71- 
110). 

Driscoll. (2005b). Constructivism. In Psychology of learning for instruction (Vol. 3, pp. 384-410). Boston: 
Allyn & Bacon. 

Driscoll. (2005c). Radical behaviorism. Psychology of learning for instruction, 29-69. 

Emert, T. (2013). ‘The Transpoemations Project’: digital storytelling, contemporary poetry, and refugee boys. 
Intercultural Education, 24(4), 355-365. 

Erol, O., & Kurt, A. A. (2017). The effects of teaching programming with scratch on pre-service information 
technology teachers' motivation and achievement. Computers in Human Behavior, 77, 11-18. 

Evancho, S. R. (2000). Critical thinking skills and dispositions of the undergraduate baccalaureate nursing 
student. 

Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive-developmental inquiry. 
American psychologist, 34(10), 906. 

Gomez Puente, S., van Eijck, M., & Jochems, W. (2013). Facilitating the learning process in design-based 
learning practices: An investigation of teachers’ actions in supervising students. Research in Science & 
Technological Education, 31(3), 288-307. 

Géc¢en, G. (2014). Dijital 6ykiileme y6nteminin 6grencilerin akademik basar1 ile 6grenme ve ders ¢galisma 
stratejilerine etkisi. Unpublished master’s thesis). University of Mugla Sitki Kogman, Mugla, Turkey. 

Heo, M. (2009). Digital storytelling: An empirical study of the impact of digital storytelling on pre-service 
teachers’ self-efficacy and dispositions towards educational technology. Journal of Educational 
Multimedia and Hypermedia, 18(4), 405-428. 

Huitt, W. (1997). Metacognition. Educational Psychology Interactive. Valdosta, GA: Valdosta State University. 


26 


Inceoglu, M. (1993). Tutum, algi, iletisim [Attitude, perception, communication]. Ankara: V Yayinlari. 

Jonassen, & Reeves, T. C. (1996). Learning with technology: Using computers as cognitive tools. InD. H. 
Jonassen (Ed.), Handbook of research for educational communications and technology (pp. 693719). 
Bloomington, IN: Association for Communications and Technology. 

Jonassen, D. H. (1999). Designing constructivist learning environments. Jnstructional design theories and 
models: A new paradigm of instructional theory, 2, 215-239. 

Kafai, Y. B., & Burke, Q. (2015). Constructionist gaming: Understanding the benefits of making games for 
learning. Educational psychologist, 50(4), 313-334. 

Ke, F. (2014). An implementation of design-based learning through creating educational computer games: A 
case study on mathematics learning during design and computing. Computers & Education, 73, 26-39. 

Keller, J. M. (1979). Motivation and instructional design: A theoretical perspective. Journal of instructional 
development, 2(4), 26. 

Keller, J. M. (1983). Motivational design of instruction. Instructional design theories and models: An overview 
of their current status, 1(1983), 383-434. 

Keller, J. M. (1987). An application of the ARCS model of motivational design. Jnstructional theories in action: 
Lessons illustrating selected theories and models, 289-320. 

Kim, B., & Reeves, T. C. (2007). Reframing research on learning with technology: In search of the meaning of 
cognitive tools. Instructional science, 35(3), 207-256. 

Kolodner, J. L., Crismond, D., Gray, J., Holbrook, J., & Puntambekar, S. (1998). Learning by design from 
theory to practice. Paper presented at the Proceedings of the international conference of the learning 
sciences. 

Kolodner, J. L., Owensby, J. N., & Guzdial, M. (2004). Case-based learning aids. Handbook of research on 
educational communications and technology, 2, 829-861. 

Korkmaz, O. (2016). The Effects of Scratch-Based Game Activities on Students' Attitudes, Self-Efficacy and 
Academic Achievement. International Journal of Modern Education & Computer Science, 8(1). 

Korkmaz, O. (2018). The effect of scratch-and lego mindstorms Ev3-Based programming activities on academic 
achievement, problem-solving skills and logical-mathematical thinking skills of students. MOJES: 
Malaysian Online Journal of Educational Sciences, 4(3), 73-88. 

Koshino, M., Kojima, Y., & Kanedera, N. (2013). Development and Evaluation of Educational Materials for 
Embedded Systems to Increase the Learning Motivation. Online Submission, 3(5), 305-313. 

Kwon, D.-Y., Kim, H.-S., Shim, J.-K., & Lee, W.-G. (2012). Algorithmic bricks: a tangible robot programming 
tool for elementary school students. JEEE Transactions on Education, 55(4), 474-479. 

Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation: Cambridge university 
press. 

Lewis, C. M. (2011). Is pair programming more effective than other forms of collaboration for young students? 
Computer Science Education, 21(2), 105-134. 

Liu, J., Lin, C.-H., Potter, P., Hasson, E. P., Barnett, Z. D., & Singleton, M. (2013). Going mobile with app 
inventor for android: a one-week computing workshop for K-12 teachers. Paper presented at the 
Proceeding of the 44th ACM technical symposium on Computer science education. 

Martin, B. (1999). In Reigeluth, CM Instructional-design theories and models Volume II (pp. 485-509). In: 

Mahwah, NJ: Lawrence Erlbaum Associates, Publishers. 

Mayer, R. E., & Moreno, R. (1998). A split-attention effect in multimedia learning: Evidence for dual 

processing systems in working memory. Journal of educational psychology, 90(2), 312. 

McCombs, B. L., & Whisler, J. S. (1997). The Learner-Centered Classroom and School: Strategies for 

Increasing Student Motivation and Achievement. The Jossey-Bass Education Series: ERIC. 

Miller, R. B., Greene, B. A., Montalvo, G. P., Ravindran, B., & Nichols, J. D. (1996). Engagement in academic 

work: The role of learning goals, future consequences, pleasing others, and perceived ability. 

Contemporary educational psychology, 21(4), 388-422. 

Moreillon, J., & Hall, R. N. (2014). Digital advocacy stories: A pedagogical tool for communicating and 

strengthening library values. Journal of Education for Library and Information Science, 100-111. 

Morelli, R., De Lanerolle, T., Lake, P., Limardo, N., Tamotsu, E., & Uche, C. (2011). Can android app inventor 

bring computational thinking to k-12. Paper presented at the Proc. 42nd ACM technical symposium on 

Computer science education (SIGCSE'1 1). 

Nelson, L. M. (1999). Collaborative problem solving. /nstructional design theories and models: A new 

paradigm of instructional theory, 2, 241-267. 

Nikou, S. A., & Economides, A. A. (2014). Transition in student motivation during a scratch and an app 
inventor course. Paper presented at the 2014 IEEE Global Engineering Education Conference 
(EDUCON). 

Omar, H. M. (2018). Enhancing automatic control learning through Arduino-based projects. European Journal 
of Engineering Education, 43(5), 652-663. 


27 


Paivio, A. (1991). Dual coding theory: Retrospect and current status. Canadian Journal of Psychology/Revue 
canadienne de psychologie, 45(3), 255. 

Papadakis, S., Kalogiannakis, M., Zaranis, N., & Orfanakis, V. (2016). Using Scratch and App Inventor for 
teaching introductory programming in secondary education. A case study. Jnternational Journal of 
Technology Enhanced Learning, 8(3-4), 217-233. 

Pellas, N., & Vosinakis, S. (2018). The effect of simulation games on learning computer programming: A 
comparative study on high school students’ learning performance by assessing computational problem- 
solving strategies. Education and Information Technologies, 1-30. 

Perkins, D. N. (1991). Technology meets constructivism: Do they make a marriage? Educational technology, 
31(5), 18-23. 

Rivet, A. E., & Krajcik, J. S. (2004). Achieving standards in urban systemic reform: An example of a sixth 
grade project-based science curriculum. Journal of research in science teaching, 41(7), 669-692. 

Roscoe, J. F., Fearn, S., & Posey, E. (2014). Teaching computational thinking by playing games and building 
robots. Paper presented at the Interactive Technologies and Games (iTAG), 2014 International 
Conference on. 

Rotherham, A. J., & Willingham, D. (2009). 21st century. Educational leadership, 67(1), 16-21. 

Sancar-Tokmak, H., & Incikabi, L. (2013). The effect of expertise-based training on the quality of digital stories 
created to teach mathematics to young children. Educational Media International, 50(4), 325-340. 

Sarier, Y. (2010). Ortadgretime giris sinavlar1 (OKS-SBS) ve PISA sonuclar1 is1g1nda egitimde firsat esitliginin 
deSerlendirilmesi. Ahi Evran Universitesi Egitim Fakiiltesi Dergisi, 11(3), 107-129. 

Saritepeci, M. (2017). Ortaokul diizeyinde dijital hikaye anlatiminin yansitici diistinme becerisi tizerindeki 
etkisinin incelenmesine yénelik deneysel bir calisma. Bartin Universitesi Egitim Fakiiltesi Dergisi, 
6(3), 1367-1384. 

Savery, J. R., & Duffy, T. M. (1995). Problem based learning: An instructional model and its constructivist 
framework. Educational technology, 35(5), 31-38. 

Skinner, & Belmont, M. J. (1993). Motivation in the classroom: Reciprocal effects of teacher behavior and 
student engagement across the school year. Journal of educational psychology, 85(4), 571. 

Skinner, B. F. (1961). The operational analysis of psychological terms. 

Spiro, R. J., Feltovich, P. L., Jackson, M. J., & Coulson, R. L. (1991). Cognitive flexibility, constructivism, and 
hypertext: Random access instruction for advanced knowledge acquisition in ill-structured domains. . 
Educational technology, 31(5), 24-33. 

Su, A. Y., Yang, S. J., Hwang, W. Y., Huang, C. S., & Tern, M. Y. (2014). Investigating the role of 
computer-supported annotation in problem-solving-based teaching: An empirical study of a S cratch 
programming pedagogy. British Journal of Educational Technology, 45(4), 647-665. 

Topalli, D., & Cagiltay, N. E. (2018). Improving programming skills in engineering education through problem- 
based game projects with Scratch. Computers & Education, 120, 64-74. 

Trilling, B., & Fadel, C. (2009). 2/st century skills: Learning for life in our times: John Wiley & Sons. 

Van Tiem, D. M., Moseley, J. L., & Dessinger, J. C. (2000). Fundamentals of performance technology: a guide 
to improving people process and performance. 

Vatansever, O., & Goktalay, S. B. (2018). HOW DOES TEACHING PROGRAMMING THROUGH 
SCRATCH AFFECT PROBLEM-SOLVING SKILLS OF 5th AND 6th GRADE MIDDLE SCHOOL 
STUDENTS? 

Vygotsky, L. (1978). Interaction between learning and development. Readings on the development of children, 
23(3), 34-41. 

Waks, S. (1995). Curriculum design: From an art towards a science: Tempus. 

Wang, X.-M., Hwang, G.-J., Liang, Z.-Y., & Wang, H.-Y. (2017). Enhancing students’ computer programming 
performances, critical thinking awareness and attitudes towards programming: An online peer- 
assessment attempt. Journal of Educational Technology & Society, 20(4), 58-68. 

Wangenheim, C. G., Alves, N. C., Rodrigues, P. E., & Hauck, J. C. (2017). Teaching Computing in a 
Multidisciplinary Way in Social Studies Classes in School--A Case Study. Online Submission, 1(2). 

Wigfield, A., & Eccles, J. S. (2000). Expectancy—value theory of achievement motivation. Contemporary 
educational psychology, 25(1), 68-81. 

Wile, D. (1996). Why doers do. Performance+ Instruction, 35(2), 30-35. 

Wing, J. M. (2006). Computational thinking. Communications of the ACM, 49(3), 33-35. 

Winn, W. (1993). A constructivist critique of the assumptions of instructional design. In Designing 
environments for constructive learning (pp. 189-212): Springer. 


28 


Using Khan Academy Mappers for Math Differentiation 


Cassandra C. Best 
Morehead State University 
2439 O’Bryan Blvd. Apt B. 

Owensboro, KY 42301 


Descriptors: Khan Academy Mappers, Math, 
Abstract 


In the 2017-18 school year, Henderson County Schools (HCS) adopted the 1:1 student-technology policy to 
incorporate a blended learning environment to increase student engagement and achievement. The HCS technology 
team applied and won the technology grant that supplied the district with Chromebooks. During our convocation, 
our theme of the year was “providing differentiation through technology.” HCS provided numerous professional 
developments for faculty that included classroom management, Google Apps for Education (GAFE), and SAMR 
models. These professional developments helped to learn how to integrate technology into the classroom. Therefore, 
the faculty was excited about the prospects of designing technology-enhanced lessons that provided differentiation 
within the class. However, the excitement was short-lived. During the parent-teacher conferences, there were 
various parents upset about the lack of differentiation within the lessons. The outrage that their children weren't 
academically challenged, especially in math. The math department was called in for a meeting after school to 
discuss some potential strategies to increase differentiation within the classroom. We were sent on more professional 
developments. 

However, these training lacked innovated techniques to provide math differentiation above the elementary 
level. Also, most of the methods were already being implemented or just impractical for our class sizes. Therefore, 
the math department was tasked with finding instructional strategies that provided differentiation for each Rasch 
Unit (RIT) band of math using technology. There is a lack of professional development and low-cost web-based 
tools that are tailored to provide differentiation in math for a large, diverse student population. It is our responsibility 
as math educators to offer rigorous and practical math instruction to all students. 

Every year, Kentucky students participate in the NWEA's Measure of Academic Progress (MAP's) testing 
for reading and mathematics. The MAP testing is usually completed in the Fall, Winter, and Spring to measure the 
academic growth of our students. Teachers are expected to provide intervention and differentiation for each student 
to achieve their growth goal. However, the traditional general classroom is comprised of twenty-five to thirty 
students with varying Rasch Unit (RIT) band scores. Therefore, Khan Academy (a free non-profit educational 
technology tool) partnered with the NWEA (creators of MAP testing) to design and personalize rigorous instruction 
for each RIT band in each of the tested categories on the MAP testing. 

Khan Academy Mappers is a math classroom-tested instructional strategy for differentiation. This learning 
technology consists of instructional videos, lessons, and interactive whiteboard for students to practice math skills at 
their own pace. Each lesson includes four to eight rigorous problems that span from concrete based math drills to 
real-world application word problems. Students are encouraged to self-learn and use the resources provided to 
complete each lesson at 100% accuracy. Additionally, Khan Academy offers students remedial lessons to complete 
to activate prior knowledge if needed. Teachers are categorized as Instructional Coaches on Khan Academy and, 
therefore, can monitor each student and their progress on completing lessons. 


Description: 


The roundtable session introduces the technological tool of Khan Academy Mappers and its instructional 
use to differentiate math in a blended classroom environment. Khan Academy Mappers is a joint endeavor with the 
Northwest Evaluation Association (NWEA) that created personalize lessons that catered to each Rausch Unit (RIT) 
band. The discussion will provide instructional strategies and a handout detailing the steps to set-up student scores 
and data-tracking examples. 
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Background of standards: 
According to the National Council of Teaching Mathematics (NCTM), the three primary principles and standards 
are the following: 

e Equity: “Excellence in mathematics education requires equity-high expectations and strong support for all 
students” (NCTM, p.2, n.d.) 

e Learning: “Students must learn mathematics with understanding, actively building new knowledge from 
experience and previous knowledge” (NCTM, p.2, n.d.) 

e = Technology: “Technology is essential in teaching and learning mathematics; it influences the mathematics 
that is taught and enhances students’ learning” (NCTM, p.3, n.d.) 

e Integrating Khan Academy Mappers ensures the standards are being implemented for all math students 
using the best instructional strategy to incorporate personalized lessons for each RIT band that is 
customized to each student's RIT scores. 

e The Khan Academy Mappers aligns with the Common Core Standards and mathematical practices for each 
grade level. Therefore, students are completing rigorous coursework per the Common Core Standards 
guidelines. 


Significance to teachers: 

The Professional Growth Plan (PGP) is the Kentucky teacher accountability program that is used for annual 

evaluation. Some of the primary components on the PGP are the following: 

Domain 1—Planning and Preparation: 1D- Demonstrating Knowledge of Resources. 

Domain 2— Classroom Environment: 2B- Establishing a Culture of Learning. 

Domain 3— Instruction: 3C- Engaging Students in Learning and 3D- Using Assessment in Instruction. 
Domain 4 — Professional Responsibilities: 4A- Reflecting on Teaching and 4B-Maintaining Accurate Records. 

The clusters stated above are some of the units of measurement that counts as teacher's annual evaluations. 
The integration of Khan Academy Mappers is a great educational tool that will aid teachers in achieving 
professional growth while increasing student growth scores on the measures of academic progress (MAPs) testing. 

Moreover, Khan Academy Mappers provides math teachers an instructional strategy to differentiate math 
for different ability levels for a large classroom. Most intervention courses and programs require small group 
instruction. 

However, Khan Academy is a technological tool that aids teachers in differentiating math instruction and 
provide rigorous coursework for students of multiple ability levels at the same time. Instructional time is precious, 
and teachers are required and expected to teach the curriculum while simultaneously personalize instruction to 
increase student learning outcomes. Khan Academy Mappers is a tool that will aid in the teacher's instructional 
design to achieve professional goals without sacrificing precious instructional time. 


Domain 1—planning and preparation: 1d- demonstrating knowledge of resources 

The subdomain demonstrating knowledge of resources is an easy component to reach an accomplish status 
by the end of the year. The subdomain requires that educators know of the available resources that can be integrated 
into the classrooms to increase student learning outcomes. One of the primary options to fulfill the criteria is to 
attend professional developments or conferences that will enhance awareness of new resources or new instructional 
strategies to incorporate resources in the classroom. Another option to fulfill the criteria is the effective planning of 
the resource to integrate into the classroom, then share the implementation and results with colleagues and parents. 

Henderson County provided a technology professional development at the end of every school year. At one 
of the sessions, the math department chair, Stacey Hyslop, introduced Khan Academy’s new beta feature Khan 
Academy Mappers. The tool was a cooperative endeavor from NWEA and Khan Academy to compile a lesson for 
each RIT band in each of the math domains for elementary and middle grades. The focus of the presentation is the 
introduction of a technological tool that would be useful for advanced math students that wanted to challenge 
themselves and increase the growth scores on the MAPs testing. Also, to provide another valuable resource for the 
educational tool kit. 

After the session, additional research was needed to learn how to implement the resource into weekly 
instruction effectively and how to use the tool for students of different math abilities. The decision to use Khan 
Academy Mappers, the next school year as a differentiation tool to increase student growth scores, and 
communicating with parents about the importance of Khan Academy Mappers tool in the classroom will aid in 
demonstrating knowledge of the resource. 
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Domain 2— classroom environment: 2b- establishing a culture of learning 

Teachers are held accountable for establishing a culture of learning in the classroom. Some strategies to 
ensure a positive learning environment is to set high expectations for students, explain the "why," and encourage 
parent involvement. Setting high expectations is critical for establishing a culture of learning. Students need to know 
what is expected of them so they can strive to achieve the goal. Also, explaining the "why" it is essential and "why" 
is it necessary to learn math is critical to increasing student engagement and motivation to complete the math 
assignments, especially for difficult concepts. 

Therefore, when introducing the Khan Academy Mappers, it is essential to instill high expectations of 
independent thinking, critical thinking, and self-learning for students. Students need to learn how to think for 
themselves. Independent thinking is such a hard concept to develop. However, when students are taking the 
standardized test or trying to solve a problem, they need to be equipped with the problem-solving strategies that will 
help in making the right choice. Students will not always be told how to solve the problem, and therefore, they need 
critical thinking skills to identify new techniques to solve problems or find unique solutions. Also, students need to 
learn how they best learn a new concept. As a middle school educator, a part of the job is ensuring students are 
ready for the next stage in their school career. In high school and college, students will have the expectations to 
retain the knowledge the best way for them and are expected to transfer or apply the knowledge in everyday 
situations and on academic assessments. 

Moreover, explaining the "why" is just as important. Students need to know why they must learn these 
skills. So, when introducing Khan Academy Mappers as a new formative assessment tool to complete during class 
and at home, it is crucial to justify the reasoning. In the Henderson County School district, students are enrolled in 
specific math courses based on their MAPs scores. If the goal for the student is to participate in the advance or 
double advance math courses, they will need to reach their growth score and score a distinguish rating on MAP 
testing. If the students do not want to take an extra math course as intervention, they will need to meet their growth 
score and grade level benchmark. So, explaining the facts to the students piques their interest in learning more about 
growth, benchmark, and the novice, apprentice, proficient, and distinguished scores. Also, explaining what growth 
means and how it works is beneficial for students to visualize their goal at the end of the school year. 

As previously mentioned, setting high expectations for students is necessary for establishing a culture of 
learning. Therefore, giving students an arbitrary point to achieve by the Spring of the school year aids in a concrete 
goal for students to complete. For example, in my classroom, I tell all students that growth is six points above their 
Spring score. I will also tell students that the six points are divided by the three MAP tests that they take throughout 
the school year, so two points for Fall, two points for Winter, and two points for Spring. So, if a student previously 
scored a 225 in the Spring of last school year, by the Spring of the current school year, students must reach a 231. 
Alternatively, for the students that score higher such as a 250, their goal by the end of the Spring of the current 
school year is 256. Most students will gasp and start to think it is impossible. Yet, it is not. That's when I introduce 
Khan Academy Mappers. Letting the students know that the Khan Academy Mappers is a site that will provide 
personalized instruction within their focus areas will help them reach their growth goal since the NWEA that wrote 
the MAP testing is the same organization that helps create the Khan Academy Mappers. Try giving the scenario of 
basketball players need to attend and give 100% at practice, so they can play the game to win. It is the same; 
students will need to participate and give 100% to completing the lessons to achieve the goal during the MAP testing. 

Furthermore, the Khan Academy Mappers will produce a productive struggle for the students. The lessons 
are personalized within their RIT bands; some of the lessons may prove to be challenging or new to the students, 
which is why it is crucial to encourage parent involvement. Communicating with parents about the importance of 
Khan Academy Mappers and providing additional resources that will help students to learn new math concepts 
outside of the classroom is critical for success. Students need to know that the parents and teachers are on the same 
page for their education to help establish the culture of learning. 


Domain 3— instruction: 3c- engaging students in learning and 

Integrating Khan Academy Mappers in class is an essential component for student engagement. Scheduling 
a block of instructional time for students to only work on Khan Academy Mappers will help in instilling the 
importance of Khan Academy Mappers and time for the students to work on new skills that may need some 
guidance from the teacher. In my classroom, every Friday during their second math block is called Khan Academy 
Friday. Students are in a routine to enter the class, log in to their Khan Academy accounts and start working on that 
nine weeks strand (1% nine weeks Operations and Algebraic Thinking, 2" nine weeks Real and Complex Numbers, 
3 nine weeks Geometry, and 4" nine weeks are Statistics and Probability). However, the setup for students can 
vary with each teacher; I use this set-up as it aligns with my curriculum map, and students are all working on the 
same strand at the same time. 
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Domain 3— instruction: 3d- using assessment in instruction 

Furthermore, I use Khan Academy as a formative assessment for student grades. Some students need that 
extra motivation for completing lessons, especially for those students that have over fifty lessons to complete within 
a strand due to their RIT band. So, assigning students a set of lessons to complete within the nine weeks is beneficial 
to prevent student shut-down. For instance, in my classroom, students are required to complete at least ten lessons 
each nine weeks. Favoring student choice as another engagement factor is beneficial, so I allow students to choose 
which ten they want to complete within that nine-week strand. To encourage students to complete additional lessons, 
I offer classroom rewards such as candy, extra credit, or tech time for students that complete five extra lessons, ten 
extra lessons, or fifteen extra lessons, respectively. The reward factor is beneficial for students that need extrinsic 
motivation to complete the Khan Academy Mappers. 

Moreover, Khan Academy Mappers permits students and teachers to update MAP scores to gain new 
lessons as the students continue to improve on the MAP test. The updating feature on the Khan Academy site is 
beneficial to continually personalize lessons for students as they continue to grow and increase their scores on MAP 
testing. 


Domain 4 — professional responsibilities: 4a- reflecting on teaching 

Reflecting on teaching is an essential skill for every teacher. Learning and understanding what worked well 
in the classroom and adjusting throughout the year to lessons that need refinement are imperative. When integrating 
Khan Academy Mappers, it is vital to observe student behaviors, attitudes, and engagement levels to identifying 
each student's zone of proximal development (ZPD). The role of the teacher is to facilitate the Khan Academy 
Mappers, where the lessons generate productive struggle. However, if lessons are too complicated, students will 
become frustrated or anxious and not succeed in completing the lessons. Also, a negative connotation will be 
attached to Khan Academy Mappers. 

Exceeding the ZPD is not the objective of Khan Academy Mappers. So, it is essential to reflect on teaching, 
to adjust the number of lessons, assign particular lessons that the students will need for the current grade level, 
adjust percentages achieved for specific lessons, or introduce or reteach strategies to help students understand the 
concept. For instance, I had a student that scored above grade level on the MAP test. Once the raw score for each 
strand was entered in Khan Academy Mappers, it generated lessons that were high school level. After careful 
observation, I conferenced with the student to identify some lessons that were grade level within the strand and some 
challenging lessons that I adjusted the percentage achieved. The strategy of reflecting on teaching was essential to 
identify what lessons are necessary for optimal growth in the eighth grade and which lessons will challenge the 
student's math abilities. 


Domain 4 — professional responsibilities: 4b-maintaining accurate records. 

Khan Academy Mappers does not have a teacher monitor screen unlike the regular classroom version of 
Khan Academy. Therefore, it is vital to maintain accurate records of student's completed lessons and students’ 
scores on the MAP test. Having a history of students MAP scores from spring of the last academic school year, fall, 
winter, and spring of the current school year is critical to use as students prepare for the MAP test to know what 
their goal to reach, to let the teacher know if the student is succeeding, and to provide as documentation for PGP. It 
is easy to create a data sheet for each class using Google Docs. On Google Docs, I created a table with the title 
listing the class period underneath the title in the right column is a list of the student’s name, the next column has the 
students last year's spring MAP score. The next column is the fall score, next column is the winter goal, the next 
column is the winter score, the next column is the spring goal score, the next column is the spring score, and the last 
column is the student's novice, apprentice, proficient, or distinguish level based on their spring MAP score. Figure 1 
is a snapshot of a template I created to use for my classroom and that I shared with colleagues to use for their class. 
It is color coated to help make the document easy to read. As a result, I let the students know if they completed their 
growth goal through a celebration. Students that meet their goal will have the names on a poster, have their choice of 
soft drink, and a bag of chips. Again, reflecting on teaching at the middle school level, some students need extrinsic 
motivation to fulfill their potential and engage in learning. 
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MAP’s Growth Test 19-20 


Figure 1. MAP's Growth Test Snapshot 


Furthermore, our math department chair Stacey Hyslop created a document for students to track their data 
and goals. The data-tracking sheet is a great resource to use for students to have a visual and a reminder of their 
goals that they need to achieve. The tracking sheet also encourages students to take ownership of their learning. I 
usually complete the top half of the sheet for the students. The bottom half displays the Khan Academy Mappers 
website and the strands for the fall, winter, and spring. Figure 2 displays a snapshot of the student data-tracking tool 
for the 2019-2020 school year with a few modifications that I made to suit my classroom and students. 


8th grade Math MAP Scores 2019-2020 


MathGoal =| Fal ~—| Winter =| Spring’ 
Distinguished =| 249-350 251-350 253-350 
Proficient =| 230-248 233-250 235-252 


210-229 213-232 215-234 
100-209 100-212 100-214 


Name: 


Spring-2019 

Goal Fall(2019) 

Fall-2019 

Goal Winter-2020 

Winter-2020 

Goal Spring-2020: 

Spring-2020 

KHAN ACADEMY MAPPERS: khanacademy.org/mappers 

Fall Strands: Operation and Algebraic Thinking 
Real and Complex Numbers. 
Geometry 
Statistics and Probability. 


Figure 2. Snapshot of Student Data-Tracking Sheet 
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Khan Academy Mappers in classroom 


Implementation in the classroom 

Henderson County North Middle School, typical schedule is two blocks of math, two blocks of English and 
language arts (ELA), Cadets in Action (CIA), and two blocks of electives. Therefore, I will have three groups of 
students twice a day. In a typical math class, the morning session consists of daily instruction, and the afternoon 
session is review and practice. 

After the first two weeks of school, I will gather the NWEA MAP data for each of my students and separate 
the data by class. Once each class is divided, I would go through the students’ score sheets and write their Spring 
Goal. During each afternoon session, I will introduce Khan Academy Mappers and explain the "why" for 
completing the Mappers. Then, I would model the instructions for the students to complete. On the smartboard, I 
will display the Khan Academy Mappers website, and show how students will log in and put in their scores. At this 
step, students may start to feel overwhelmed based on how many lessons they will need to complete lessons usually 
range from one to one hundred and fifty. Therefore, to reduce the feeling of panic, I will assign a set amount of 
lessons for the students to complete each term. Usually, I will assign ten lessons (each lesson will be worth 10 
points) for the entire nine weeks, and the students will have a designated day to work on during class time. Those 
designated days are called "Khan Academy Fridays". In addition, I assign the students the same strand to work on 
during the nine weeks. Khan Academy Mappers recommends for students to work on the strand that scored the 
lowest on first, but I like my classes to be uniformed and for the Khan Academy Mapper lessons to align with the 
units that I am teaching. 

The Khan Academy Mappers is a formative assessment grade. Each lesson is worth ten points for a total of 
a hundred points. Students will have the entire nine-weeks to complete the assignments. Students have the option to 
stay after school one day of the week to work on the Khan Academy Mapper lessons. Students must score at least a 
seventy percent to obtain full points. Since the Khan Academy Mappers is still in Beta testing, the teacher can not 
monitor student progress on the Khan Academy Coach screen. Therefore, students must take screenshots of 
completed lessons, paste screenshots on a Google document, and upload the Google document to Google Classroom 
for submission. Students are rewarded for completing additional lessons. For every five other lessons students 
complete they will receive PBIS points and candy, for every ten other points students will earn three extra credit 
points on lowest formative assessment, and if the student completes fifteen or more additional lessons, they will 
receive ten minutes of tech time during the afternoon session of math class. 

As illustrated in table one, units and the timeframe for every nine weeks are listed with the corresponding 
strand. Some strands will occasionally overlap with some of the other units, but the majority are aligned. One of the 
major units in eighth-grade math is the slope. The slope is practice in operations and algebraic thinking strand, so it 
is a great conversation starter and introduction to the slope. Students that practice slope during the operations and 
algebraic strand typically develop an understanding of how to identify and calculate slope on a concrete level. Thus, 
the group of students that complete slope lessons, have fewer difficulties during the slope unit and most likely 
become peer tutors throughout the unit. However, it is necessary to practice geometry during the third nine weeks 
since the KPREP is completed during the fourth nine weeks, and geometry is one of the primary math strands tested. 


Table 1. 
Eighth Grade Math Curriculum Map 
Nine Weeks Thematic Units Khan Academy Mappers Strand 
First Nine Weeks Unit 1: Exponents Operations and Algebraic 


August through October (before 
fall break) 


Unit 2: Rational and Irrational 
Numbers 


nit 3: Solving Equations (first 
half) 


Cc 


Thinking 


Students complete ten lessons of 
their choosing from this strand. 


Second Nine Weeks 
October (after fall break) 
through December 


Unit 3: Solving Equations (second 
half) 

Unit 4: Volume 

Unit 5: Slope 


Real and Complex Numbers 


Students complete ten lessons of 
their choosing from this strand. 
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Third Nine Weeks Unit 6: Pythagorean Theorem Geometry 
January through March Unit 7: Systems of Equations 
Unit 8: Functions Students complete ten lessons of 
Unit 9: Scatterplots their choosing from this strand. 
Fourth Nine Weeks Unit 10: Transformations Statistics and Probability 
April through May Unit 11: Angles 
Students complete ten lessons of 
their choosing from this strand. 
MAP correlation to KPREP 


A significant factor for teachers and students is to understand the MAP score correlation with the KPREP. 
The NWEA website offers multiple tables to display various types of correlations. The correlation that I print out for 
each of my students and review are the KPREP scores and the spring MAP scores. Therefore, starting with the end 
in mind, the students will identify their goal for KPREP and the score they will need to achieve on the spring MAP 
test to ensure that they are on the right track. Figure 3 shows the correlation tables of the KPREP scores and 
percentiles with the corresponding spring MAP testing scores. 


K-PREP 
Grade Level 1 Level 2 Level 3 Level 4 
Novice Apprentice Proficient Distinguished 
3 100-191 192-209 210-233 234-300 
4 100-193 194-209 210-228 229-300 
5 100-191 192-209 210-228 229-300 
6 100-190 191-209 210-230 231-300 
7 100-191 192-209 210-230 231-300 
8 100-191 192-209 210-231 232-300 
MAP 
as ve u | — —_ 3 meine 
Novice Apprentice Proficient Distinguished 
RIT %ile RIT %ile RIT %ile RIT %ile 
3 100-191 1-19 192-204 20-53 205-217 54-84 218-350 85-99 
4 100-198 1-15 199-212 16-47 213-225 48-78 226-350 79-99 
5 100-202 1-12 203-220 13-47 221-235 48-80 236-350 81-99 
6 100-205 1-11 206-223 12-45 224-238 46-78 239-350 79-99 
7 100-212 1-18 213-230 19-54 231-245 55-83 246-350 84-99 


8 100-214 1-19 215-234 20-57 235-252 58-87 253-350 88-99 


Notes. 1. %ile=percentile. 
2. Bolded numbers indicate the cut scores considered to be at least “proficient” for accountability 
purposes. 


Figure 1. Correlation of KPREP and spring MAP scores (NWEA, 2017) 


Another exciting factor that teachers can use as a method of reflection and aid in completing the mid- 
reflection is the correlation tables of the KPREP scores and the fall and winter MAP scores. Our school district, 
MAP test in the fall, winter, and spring. Therefore, the added piece of data is beneficial to identify students are 
falling behind, on-track, or exceeding the expected goals. Figure 4 is a great conference and communication tool to 
use with students one-on-one to encourage students to keep working hard or to identify the potential areas and 
strategies that students may need additional guidance. 
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K-PREP 


Grade Level 1 Level 2 Level 3 Level 4 
Novice Apprentice Proficient Distinguished 

3 100-197 198-209 210-225 226-300 

4 100-196 197-209 210-226 227-300 

5 100-197 198-209 210-225 226-300 

6 100-198 199-209 210-226 227-300 

7 100-198 199-209 210-225 226-300 

8 100-198 199-209 210-224 225-300 

MAP FALL 

Level 1 Level 2 Level 3 Level 4 

aes Novice Apprentice Proficient Distinguished 


RIT %ile RIT %ile RIT %ile RIT %ile 
100-178 1-26 179-190 27-55 191-202 56-81 203-350 82-99 
100-187 1-24 188-199 25-53 200-213 54-83 214-350 84-99 
100-193 1-21 194-206 22-52 207-221 53-85 222-350 86-99 
100-200 1-24 201-210 25-48 211-225 49-83 226-350 84-99 
100-203 1-23 204-214 24-50 215-228 51-82 229-350 83-99 
100-208 1-28 209-220 29-58 221-234 59-86 235-350 87-99 

MAP WINTER 
Level 1 Level 2 Level 3 Level 4 
Novice Apprentice Proficient Distinguished 

RIT %ile RIT %ile RIT %ile RIT %ile 
3 100-186 1-27 187-197 28-54 198-208 55-80 209-350 81-99 
4 100-193 1-24 194-205 25-55 206-218 56-84 219-350 85-99 
5 100-198 1-21 199-210 22-51 211-224 52-84 225-350 85-99 
6 
7 


on ou & WwW 


Grade 


100-204 1-25 205-213 26-48 214-227 49-81 228-350 82-99 
100-206 1-24 207-217 25-51 218-229 52-79 230-350 80-99 


8 100-211 1-31 212-221 32-56 222-235 57-85 236-350 86-99 
Notes. 1. %ile=percentile. 
2. Bolded numbers indicate the cut scores considered to be at least “proficient” for accountability 
purposes. 


Figure 2. Correlation of KPREP and fall and winter MAP scores (NWEA, 2017) 
Results 


Khan Academy Mappers provided differentiation to all students on various RIT levels. Parents were 
satisfied with the level of rigor that the students were practicing math at home and in the classroom. The students 
were motivated to gain new skills that increased their MAP scores and understanding of math concepts. 

Also, Khan Academy provided the flexibility of updating the student's scores after each MAP test to allow 
students to continue to improve their MAP scores. At the end of the year, most of the students surpassed their goal 
and growth score on the spring MAP testing. The effective use of Khan Academy Mappers differentiation narrowed 
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the achievement gap within the school district. 

During the fall of the 2018-19 school year, KPREP scores were released, and the math department was 
praised in the increase test scores of the students. The HCS middle school math scores were 8" in the state with a 
score of 66.3, which surpasses the state average of 47 by 19.3 points. 
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Abstract 


This study presents a social network analysis of the keywords attached to articles published in the 
Turkish Online Journal of Distance Education (TOJDE), a prominent journal in the field of open and distance 
learning. The social network analysis applied was based on a data mining and analytics approach. A total of 
1120 keywords from 784 selected articles constituted the sample of the study. The keyword analysis revealed 
that the articles published in the TOJDE largely focused on technology-related issues, suggesting that the issues 
related to educational technology are a popular research area. However, the analysis also found that there was an 
imbalance between educational technology-related topics and pedagogy-related topics, a critical issue that needs 
to be further considered. 


Introduction 


For any discipline, it is important to understand the empirical evidence that has been gathered to realize 
its origin and future direction. Past experiences and empirical contributions are helpful in providing a sense of 
the continuum from the past to the present and the future. From this point of view, researchers in the field need 
to analyze and synthesize research trends to gain a better vision for future implications. Moreover, it is crucial to 
define the research areas of distance education and to fill the content of these areas with scholarly works and 
research-based evidence. Guided by this understanding, this paper aims to explore patterns by conducting a 
social network analysis of the keywords in articles published between 2000 and 2015. 


According to Bozkurt and Akgun-Ozbek (2015), identifying research patterns is a critical task, insofar 
as these patterns can serve as a guide to researchers in the field. In this regard, much research has examined the 
scholarly landscape in the field of distance education from a global perspective (Amoozegar, Khodabandelou, & 
Ale Ebrahim, 2018; Berge & Mrozowski, 2001; Bozkurt et al., 2015; Cakiroglu, Koko¢, Gokoglu, Oztiirk, & 
Erdogdu, 2019; Hauser, 2013; Koble & Bunker, 1997; Lee, Driscoll, & Nelson, 2004; Orellana & Nethi, 2019; 
Rourke & Szabo, 2002; Tuncay & Uzunboylu, 2010; Wong, Zeng, & Ho, 2016; Zawacki-Richter & Naidu, 
2016; Zawacki-Richter, Alturki, & Aldraiweesh, 2017; Zawacki-Richter, Backer, & Vogt, 2009; Weller, Jordan, 
DeVries, & Rolfe, 2018) as well as from local perspectives (Asdaque & Rizvi, 2019; Asdaque, Rizvi, Jumani, & 
Ahmed, 2018; Bozkurt et al., 2015; Durak et al., 2017; Gékmen et al., 2017; Horzum, Ozkaya, Demirci, & 
Alpaslan, 2013; Panda, 1992; De Olivera Neto, & dos Santos, 2010; Ritzhaupt, Stewart, Smith, & Barron, 
2010). To note, similar to the aim of this study, there have been two previous studies conducted that have 
analyzed the research trends in TOJDE. The overall objective of these other studies was to identify the research 
trends in TOJDE over a specific period. 


In the first of these studies cited as being similar to the present, Latchem (2009) conducted a content 
analysis of the Notes for Editors and the articles published in the TOJDE between 2000 and 2008. The analysis 
involved determining the articles’ countries of origin, the sectors represented, and the focus and frequency of the 
topics covered. It was reported that the majority of articles originated from Asian countries, with Turkey 
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providing the largest number of contributions. There were also many papers from the Middle East, Africa, South 
America, the USA, Eastern and Western Europe, and Australia. According to Latchem, some of the papers that 
presented non-Western perspectives were particularly illuminating. The earlier articles tended to be descriptive 
or theoretical, while more recent ones examined distance education and elearning needs, policies, procedures, 
practices and outcomes through a quantitative-experimental and qualitative-descriptive lens. The second of these 
two similar studies, which was conducted by Ozarslan, Balaban-Sali and Demiray (2012), involved analysis of 
the articles in TOJDE published between 2000 and 2010, where the focus was on research topics, methods, 
instruments, statistical methods, author numbers and their institutional affiliation. They reported that single- 
author articles constituted the most substantial proportion of TOJDE articles. It was also indicated that Turkey, 
the USA, India, Nigeria, Malaysia, Pakistan, Australia, Canada, the UK, Bangladesh, Greece, and Iran, 
respectively, were the chief contributors to TOJDE. In addition, the researchers identified research topics 
covered in TOJDE articles. Accordingly, (1) Learner and instructor experiences in online learning 
environments; (2) information about the system and program; (3) the economic, social and cultural dimension of 
distance education, and (4) pedagogical, political, philosophical, legal, ethical reflections in distance education 
were the top four research topics addressed in TOJDE. In terms of method, it was found that quantitative studies 
far outweighed the qualitative and mixed studies. The study further revealed that document analysis was the 
most frequently used research instrument, which was followed by surveys, scales and interviews. In terms of 
data-analysis methods, descriptive statistics, content analysis, variance analysis and t-test were reported to be the 
most frequently used analysis methods. 


Methodology 


Research design 

The present study is a review study that has aimed to examine (1) keywords of articles and (2) the 
publication network in cited references in articles published in the TOJDE. To conduct this review, social 
network analysis, a data mining and analytics approach, was applied to arrive at a synthesis. 


Social Network analysis 

Social network analysis provides a powerful way to map, summarize, and visualize networks and to 
identify critical nodes that occupy strategic locations and positions within the matrix of links (Hansen, 
Shneiderman, & Smith, 2010; Scott, 1988). In this research, a social network analysis was used to examine the 
patterns that emerge from the keywords in selected articles published in the TOJDE. 


Sample: Articles published in TOJDE 

TOJDE is an open access journal published by Anadolu University in Eskisehir, Turkey. For this study, 
all the articles published in the TOJDE between 2000 and 2015 were surveyed (N=784) (Table 1). Book reviews 
and editorials were excluded from the sample. 


Table 1. Number of articles published per year in the TOJDE (Volumes 1-16) 


Year No. of issues No. of articles Year No. of issues _No. of articles 
2000 2 12 2008 4 54 
2001 2 6 2009 4 57 
2002 4 27 2010 4 51 
2003 4 23 2011 4 71 
2004 4 34 2012 4 90 
2005 4 37 2013 4 93 
2006 4 56 2014 4 75 
2007 4 51 2015 4 47 
Total 784 
Limitations 


First and foremost, since this study analyzed only those articles published in the TOJDE, it provides but 
a partial view. Hence, the findings derived from this study should be considered as complementary to those from 
other studies and be used for purposes of comparison and contrast. Second, though Open and Distance Learning 
(ODL) and Distance Education are terms similar in scope, they each have important unique features. However, 
throughout the research, these terms were used interchangeably due to the fact that their distinction was not so 
clear in the articles published in the TOJDE. 
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state of TOJDE publications. 


Table 2. List of the most frequently used keywords in articles published in the TOJDE between 2000 and 2015. 


Findings 


The analysis revealed there to be a total of 1120 keywords. The top 150 keywords with a minimum 
frequency of two are provided in Table 2. Descriptive statistics were used to provide insight into the current 


# Keyword f % # Keyword f % 
1} Distance Education 175 15.6 76|Student 4 0.4 
2|E-learning 75 6.7 77|Student Support Services 4 0.4 
3|Distance Learning 45 4.0 78|Students 4 0.4 
4|Online Learning 32 2.9 79| Teachers 4 0.4 
5|Higher Education 28 25) 80] Television 4 0.4 
6|Blended Learning 22 2.0 81} Training 4 0.4 
7 {Internet 19 ed 82[Virtual Classroom 4 0.4 
8|Education 18 1.6 83] Adult Learners 3 0.3 
9|Open And Distance Learning 22 2.0 84] Australia 3 0.3 
10|Turkey 15 1.3 85|Blogging 3 0.3 
11] Evaluation 14 i) 86|Blogs 3 0.3 
12|Learning 14 1,3) 87|Challenges 3 0.3 
13|Mobile Learning 19 ed 88|Computer Anxiety 3 OFS) 
14]Technology 13 1,2) 89| Cooperative Learning 3 0.3 
15] Attitude 1 1.0 90|Critical Thinking BI 0.3 
16|Online Education 11 1.0 91|Curriculum 3 0.3 
17|Web 2.0 1 1.0 92|Digital Natives 3 0.3 
18] Assessment 10 0.9 93|EFL Learning 3] 0.3 
19] Web-based Learning 10 0.9 94|Educational Technologies 3] 0.3 
20] Instructional Design 9 0.8 95|English Language Teaching 3] 0.3 
21|Teacher Training 9 0.8 96|Faculty 3] 0.3 
22|Interaction 8 0.7 97|Faculty Development 3] 0.3 
23|Learner Support 8 0.7 98] Flexible Learning 3] 0.3 
24|Multimedia 8 0.7 99|Gender 3] 0.3 
25]Online 8 0.7 100|Hellenic Open University 3] 0.3 
26|Professional Development 8 0.7 101 | Individual Differences 3] 0.3 
27|Teacher Education 8 0.7 102|Information 3] 0.3 
28|Allama Iqbal Open University ui 0.6 103] Instructional Technology 3] 0.3 
29 Attitudes il 0.6 104 |Internet-based Education 3] 0.3 
30|Collaboration 7 0.6 105|Knowledge Management 3] 0.3 
31}Collaborative Learning a 0.6 106|Language Learning 3] 0.3 
32|Globalization 7 0.6 107|Learning Environments 3] 0.3 
33|Learning Management System 7 0.6 108 | Lifelong Learning 3] 0.3 
34|Learning Styles 7 0.6 109|Management 3] 0.3 
35|Open And Distance Education il 0.6 110|Material 3] 0.3 
36|Perception 7 0.6 111|Mathematics Education 3] 0.3 
37|Anadolu University 6 0.5 112|Mobile 3] 0.3 
38|Blog 6 0:5 113]Nursing 3] 0.3 
39|Constructivism 6 0.5 114|Online Courses 3] 0.3 
40|Educational Technology 6 0.5 115]Online Instruction 3] 0.3 
41|Information Technology 6 0.5 116]Online Professional Development 3] 0.3 
42|Motivation 6 0.5 117|Pedagogy 3] 0.3 
43| Quality 6 0.5 118|Perceptions 3] 0.3 
44 | Satisfaction 6 0.5 119|Physics Education 3] 0.3 
45|Student Satisfaction 6 0.5 120|Problem Solving 3] 0.3 
46|Academic Achievement 5 0.4 121] Quality Education 3] 0.3 
47|BOU = 0.4 122|Science 3] 0.3 
48 |Development =) 0.4 123 |Skills 3] 0.3 
49|Distance Learners 5 0.4 124|Student Achievement 3] 0.3 
50|Facebook 5 0.4 125|Sustainability 3] 0.3 
51|Moodle 5 0.4 126|Teaching 3] 0.3 
52|Open Education Faculty e) 0.4 127|The Internet 3] 0.3 
53}Open Learning 5 0.4 128} Virtual Environments 3] 0.3 
54 [Participation 5 0.4 129] Virtual Worlds 3] 0.3 
55|Social Networking 5 0.4 130]Web Based Learning 3] 0.3 
56|Teaching Practice 5 0.4 131|Web-based Education 3] 0.3 
57|Weblog 5 0.4 132|Women 3] 0.3 
58|Basic Psychological Needs Theory 4 0.4 133 |Zimbabwe Open University 3] 0.3 
59|Blog And Social Network 4 0.4 134|AIOU 2] 0.2 
60|Certificate Program 4 0.4 135|Academic Performance 2] 0.2 
61/Communication 4 0.4 136|Access 2] 0.2 
62|Connectivism 4 0.4 137|Achievement 2] 0.2 
63 | Culture 4 0.4 138|Achievement Motivation 2] 0.2 
64| Innovation 4 0.4 139|Adult Learning 2] 0.2 
65| Internet-based Learning 4 0.4 140] Africa 2| 0.2 
66|Learning Environment 4 0.4 141 |Assignment 2] 0.2 
67|Malaysia 4 0.4 142|Assignments 2| 0.2 
68|Open Education 4 0.4 143 |Audio 2] 0.2 
69|Physical Education 4 0.4 144| Awareness 2] 0.2 
70|Qualitative Research 4 0.4 145|B.Ed Programme 2] 0.2 
71|Second Life 4 0.4 146 |Bangladesh 2| 0.2 
72 |Self-efficacy 4 0.4 147 |Barriers 2] 0.2 
73 |Self-regulated Learning 4 0.4 148 |Belief 2] 0.2 
74|Social Media 4 0.4 149 [Benefits 2] 0.2 
75 |Social Presence 4 0.4 150 |Cognitive Load 2] 0.2 


40 


For instance, in Table 2, it shows that following Distance Education, learning-oriented keywords (e.g., 
elearning [#2], distance learning [#3], online learning [#4], blended learning [#6], open and distance learning 
[#9], mobile learning [#13], web-based learning [#19], etc.) occupy the upper part of the list. Another 
interesting finding from the list is the institutional affiliations related to the territorial information used as 
keywords in reference to the physical context of the papers published in the TOJDE; these keywords include 
Allama Iqbal Open University (#28), Anadolu University (#37), Hellenic Open University (#100), and 
Zimbabwe Open University (#133), as well as the keywords referring to the country itself, such as Turkey (#10), 
Malaysia (#67), Australia (#84), Africa (#140), and Bangladesh (#146). Consistent with the observations made 
by Latchem (2009) and Ozarslan, Balaban-Sali and Demiray (2012) in their studies, the descriptive information 
gives the impression that TOJDE is a local and global voice for the field, particularly for those from Asia and 
Africa. 

In order to better understand the research patterns in the articles published in the TOJDE, a social 
network analysis based on keywords used in each article was conducted. In this research, keywords were 
identified as nodes, while their relationships, which were based on co-occurrence, were identified as ties. Of the 
1120 keywords examined, a total of 130 that had a minimum co-occurrence of three were identified. From this 
identification, a visual of the keyword network was created. The network graph created represents 130 nodes 
with 499 ties among them. (Figure 1). The nodes in the network were grouped using the Clauset-Newman- 
Moore cluster algorithm, and the Grid layout algorithm was used to lay out the graph. 
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Figure 1. Social network analysis of the keywords used in the articles 
published in the TOJDE between 2000 and 2015 


The analysis was based on betweenness centrality, which demonstrates both the bridging score of the 
nodes and their strategic position. Unsurprisingly, distance education emerged as the central node and was found 
to be strongly linked to elearning, higher education, Internet, distance learning, technology, ICT, and online 
learning (Table 3). One of the most salient findings from the social network analysis conducted was the 
tendency towards excessive dependency of distance education on “technology-based” practices (e.g., Internet, 
technology, and ICT) (see Figure 1 and Table 3). 
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Table 3. The nodes with high betweenness centrality scores 


Node Degree Betweenness Centrality 
Distance Education 81 3502.0 

elearning 50 1442.2 

Higher Education 28 564.2 

Internet 22 391.0 

Distance Learning 30 390.5 

Technology 26 332.2 

ICT 29 318.0 

Online Learning 25 307.3 


Following the investigation of the nodes, the ties among the nodes were examined according to their 
weights (Table 4). In confirmation of the results of the betweenness centrality analysis, the strongest relationship 
appeared to be between distance education and elearning. This was followed by distance education and ICT, 
which was expected considering the growing influence of technological developments and capacity increase due 
to the opportunities provided by ICTs. Similarly, distance learning and elearning; distance education and 
educational technology; higher education and ICT, and, lastly, elearning and ICT emerged as significant links. 
Overall, the weights of the links showed that distance education has established strong bonds with online 
practices and educational technology, with a focus on higher education. 


Table 4. The relationship of the keywords with high weight scores. 


Node 1 Node 2 Weight 
Distance Education eLearning 21.83 
Distance Education ICT 13.67 
Distance Learning elearning 10.67 
Distance Education Educational Technology 9.00 
Higher Education ICT 7.00 
elearning ICT 5.83 


The findings from this study indicate that open and distance learning has evolved into a digital form, 
with a growing interest shown to online modes of delivery (see Table 3). This confirms earlier studies reporting 
that online learning, or elearning, has become the new normal for open and distance learning (Amoozegar, et al., 
2018; Weller et al., 2018; Wong et al., 2016) and is in line with the study by Cakiroglu et al.(2019), who 
claimed that “new pedagogical approaches and online learning designs have played a triggering role in research 
topics” (p. 1). Another interesting finding from this study is the strong relationship between ICT and distance 
education practices (see Table 4). This, as expected, stems from the accessibility opportunities provided by ICT 
(Cakiroglu et al., 2019; Lee, 2017), which provides learners flexible, open, online learning options that help to 
facilitate and increase learners’ independence and autonomy (Harasim, 2000; Canadian Council on Learning, 
2009). While there have been many opportunities that have emerged with ICT, and these opportunities have led 
the way for online learning and elearning, Lee (2017) highlighted that pedagogic developments and 
technological development have not advanced in an equal manner. The findings from this study also partly 
confirm those reported by Lee (2017), insofar as they demonstrated that pedagogic keywords neither hold 
secondary positions (Table 2) nor have lower betweenness centrality metrics (Figure 1). This may derive from 
the marketing potentials of online learning / elearning practices (Harting & Erthal, 2005) and from the conflict 
between for- profit, commercial ambitions and the core values of open and distance learning (Evans & Pauling, 
2010). However, it should be noted that the interpretation of these findings was derived from the analysis of 
only one journal and thereby provides only a partial view of the current state of open and distance learning. 
These findings should, therefore, be validated in future research. 


Conclusions 


With the aim of identifying patterns in the keyword network, this study examined articles published 
between 2000 and 2015 in the TOJDE. The findings show that TOJDE reflects research from local (e.g., 
Anadolu University) and global perspectives, where in the case of the latter, the focus is on specific 
characteristics of regional practices. This is thought to be promising in terms of providing diverse points of view 
and representing a territorially broad voice for this field. 
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Another intriguing finding from the study is the growing influence and dominance of technology-based 
practices. This is thought to be significant in terms of tracking trends in the field of distance education through 
regional journals such as TOJDE and of providing a complementary view by comparing results of similar 
studies. However, as discussed in the previous section, the pedagogical topics seem to be under the lure of 
technology-related research topics (e.g., ICT, elearning, online learning, etc.), a situation that could create a 
future bottleneck in the field. 
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Abstract 


This paper reports on the design and development of an intelligent, natural language processing tool, the 
Teacher Responding Tool (TRT), that provides response recommendations to teachers to foster consistent, content- 
specific feedback based on student cognition. Placing student ideas at the center of instructional decisions promotes 
equitable teaching. Results indicate that the TRT selected accurate recommendations and that the interface promoted 
the teachers’ thoughtful consideration of these recommendations. Future design recommendations are provided. 


Introduction 


Research in teacher education shows that placing student ideas at the center of instructional decisions is 
critical for promoting equitable student participation, achievement, and agency (NCTM, 2014). However, 
responding to students in the moment is complex. First, teachers must infer the current understanding of the student 
(Coffey, Hammer, Levin, & Grant, 2011). Second, teachers must prioritize which understandings to focus on for the 
sake of the student, the class, and the intended learning goals (Ball, 1993). Third, the delivery of the response must 
be student-specific, be given in manageable chunks, do more than highlight errors, and avoid comparisons with 
other students (Shute, 2008). Finally, teachers should ask questions that support further student discourse (Chapin, 
O’Conner, & Anderson, 2009). 

Given this complexity, providing teachers with opportunities to develop these skills is important, and there 
have been calls to develop a variety of “approximations of practice” (Grossman, Compton, et al., 2009). This paper 
describes the design and development of an intelligent tool to scaffold teachers’ skills at giving high quality, student 
specific feedback. The Teacher Responding Tool (TRT) is a natural language processing (NLP) tool grounded in 
design principles for worked examples and developing thinking skills (Clark & Mayer, 2016) that provides 
recommendations to support teachers while they respond to students. The TRT builds upon research with 
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technologies that automatically respond directly to students (Aleven & Koedinger, 2002) and that provide teachers 
with insight into student thinking (McDonald, Bird, Zouag, & Moskal, 2017). 


Theoretical Framework 


This study is informed by instructional design principles aligned with the cognitive theory of multimedia 
learning (Mayer, 2014). These research-based principles describe how to design for learning in contexts that involve 
text and images. While our design is text-based, the assumptions of this theory and many of the design principles 
that follow from it guided our design. For example, this study is aligned with the assumptions that learners have a 
limited capacity to process information, and that learners engage in active processing via selecting, organizing, and 
integrating text. We build upon those design principles that suggest limiting extraneous material, adding cues and 
highlights, using worked examples, and focusing on authentic job-relevant thinking skills (Clark & Mayer, 2016). 


Design of the Teacher Responding Tool (TRT) 


The TRT system consists of three interacting sub-systems: the training dataset creation sub-system, the 
natural language processing (NLP) recommendation engine sub-system, and the user interface sub-system. The TRT 
system is designed to be implemented within authentic learning contexts. To set-up the system for a given learning 
context, two steps need to be taken. First, the training dataset needs to be created and then used to train the NLP 
recommendation engine. The training dataset is created by consulting with the teacher users and collecting their 
prior or suggested responses to prior student explanations for the given context. Second, the TRT needs to be 
connected to the learning management system that the students will be using so that it can pull new student 
explanations after they have been written and push teacher responses back to students. When the TRT is in use, new 
student explanations are pulled from the learning management system and used to select feedback recommendations. 
The recommendations are presented to teachers via the user interface, and the teacher response is then made 
available to the students by pushing these responses to the learning management system. The TRT system is 
designed to include teachers into the feedback process and requires teacher user interaction. As such it is intentional 
that this system does not provide instantaneous feedback to students. Figure | illustrates the subsystems and overall 
system flow. 
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The Teacher Responding Tool System 
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Figure 1. The Teacher Responding Tool (TRT) system. 


Training data subsystem 

Training data creation should aim to use authentic student explanations. In our study we began with student 
explanations collected during a prior implantation of the project within the same context (i.e. same school, same 
course, similar student population). The teacher responses were created by: first, consulting with the teachers about 
how they would respond to each of the student explanations; second, drawing upon research finding for how to best 
provide formative feedback; and third, drawing upon researcher expertise to standardize the responses. This led to a 
training dataset consisting of pairs of student explanation and teacher response written in natural language text. 
Table 1 shows the first few records of training data used for one of the questions in this study. A different training 
dataset needs to be created for each question prompt. 


Table 1. A sample of the 116 student explanation and teacher response pairs in the training dataset for the question 
prompt: “How does the scale factor change the length, width and height of the object?” 


Student Explanation Teacher Response 


It gets bigger Well done for noticing that there is a change, but what is “it’”? Is the change 
always larger? What happens when the scale factor is 10 or 0.5? 


It makes the size bigger. the Well done for noticing that the dimensions change in response to the change in 


scale factor times by the scale scale factor and that change is "multiplicative", but is this change always 
bigger or increasing? 
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It multiplies the original length Well done for noticing that the length changes in response to the change in the 
by the number of scale factor. scale factor and that the change is “multiplicative”, but what about the other 
dimensions — width and height? 


Length and width and height Well done for noticing that the dimensions changes and that the change is 
multiplied by the 3 "multiplicative", but what change in the scale factor causes these changes? 
What happens to the dimensions when the scale factor is 0.5? 


The bigger the scale factor the Well done for noticing that the dimensions become bigger as the scale factor 

bigger the dimensions get. becomes bigger, but what happens when the scale factor is made smaller rather 
than bigger? 

The new measurements are way Well done for noticing that the size changes, but does the size always increase 

higher than the old in response to change in the scale factor? What happens when the scale factor 

measurements, its shape has = 0.5? What is "it"? Expand on your answer. 

increased. 


We developed training datasets for each question that were on average about 100 records (pairs of student 
explanation and teacher response) long. Generally, larger datasets are known to improve the recommendation engine 
performance. However, factors such as the generality the question being asked and the number of different 
recommendations to choose from also impact how large the training dataset should be (Burrows, Gurevych, & Stein, 
2015; Zehner, Salzer, & Goldhammer, 2016). At the same time, the advantages of larger datasets are offset by the 
time and cost involved in creating them. Based on these considerations, we concluded that about 100 records 
represented a reasonable dataset size. 


NLP recommendations engine subsystem 

The NLP recommendation engine consists of two components: the preprocessing of student explanations, 
and a tf-idf (term frequency-inverse document frequency) model (see Figure 2). The recommendation engine is 
initialized using the training dataset. Each of the student explanations in the training dataset are preprocessed and 
then used to build the tf-idf model. The recommendation engine is used by querying the model with new student 
explanations that have been preprocessed in the same way, and the recommendations selected are outputted. 


Training Data 


Preprocessing 


New Student 
Explanation 


Feedback 
Recommendations 


Figure 2. The natural language processing (NLP) recommendations engine subsystem. 


Preprocessing involves separating a student explanation into individual words (tokenizing) and then 
applying automatic spelling correction to each word. Each word is then converted to its stem, so that words like 
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“increases,” “increasing,” and “increased” all become “increase.” Finally, common high frequency words, or stop 
words, such as “a,” “the,” and “at” are removed. Tokenizing and stemming are performed using the Python n/tk 
package, the spelling correction is performed using the Python autocorrect package, and the stop word list included 
in the Python sklearn package is used. 

The model is built by creating vector representations of each preprocessed student explanation. Weights for 
each word are determined using tf-idf (term frequency-inverse document frequency) which assigns higher weights to 
words that occur less often in the training dataset (Zehner, Salzer, & Goldhammer, 2016). The model is queried by 
finding how similar a new student explanation is when compared to each of the training dataset student 
explanations. Similarity is determined using cosine similarity, i.e. the cosine of the angle between vector 
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representations of student explanations. The teacher feedback in the training dataset that corresponds to the student 
explanations that are most similar to the new student explanation are then examined and the top three unique teacher 
feedback responses are selected as the recommendations. The model building and querying was performed using 
TfidfVectorizer within the Python sklearn package. 


User interface subsystem 

The TRT user interface is an interactive webpage that presents teachers with the question prompt, the 
student explanation, and a teacher-response field (see Figure 3). The three TRT-recommended responses are shown 
in a column on the right side of the screen. When mouse-clicked, the text of the recommendation is copied to the 
teacher-response field, and any text in the teacher-response field can be edited. This allows teachers to use the 
recommendations without making changes, customize a recommendation, take parts of different recommendations, 
or ignore the recommendations and write their own response. 

The layout of the user interface was designed to reduce the extraneous cognitive load that result from 
navigating the page, allowing the teachers to focus their working memory on considering of the recommendations. 
Recommendations were presented near to the student explanations, no scrolling was needed to navigate the page, 
clicking recommendations copy-and-pasted the text, and important text was subtly highlighted. 

The decision to present three recommendations was based on a trade-off between providing teachers with 
enough recommendations to promote the thoughtful consideration of different perspectives, but not too many so as 
to make the cognitive load of the task high and overwhelming. For each recommendation, teachers were expected to 
read the text, consider it in relation to the student explanation, and compare it with other recommendations. From 
this perspective, and considering the length of the text of the recommendations, four recommendations were 
considered by the designer to be the upper limit, two a lower limit, so three were chosen. 


How does the scale factor change the length, width and 


ee Feedback 


Wel done tor notong that the a 
ais clits Nadeaiiaan Recommendations 


New Student 


Explanation changed in response to the change 
in the scale factor, but state how 
much ig the change? If the scale 
factor is 10 oF 0.5, what would be 
the change in leegt, width and 
height of the candy? 

Write your feedbeck here. Or click @ 
eneimmaeiiaes Well Cone for noticing that ine size 
Changes and that the change is 
‘Well done for noticing the scale “mumtiolicative’, but does the size 
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Neight using the scale factor. Is this 20 change in the scale factor? What 
Teacher charge sways ncressing the happens when the scale factor « 
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Response the scale tactor was 0.5? How could 


Figure 3. The user interface subsystem. The teacher user interface displays the question prompt for which the 
training data was collected (top), the new student explanation (top left, in blue), the three recommendations from the 
NLP recommendations engine, and space for the teacher response (bottom left). 
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Method 
Research Questions 


To assess the effectiveness of the tool we asked: 
1. How accurately does the TRT select recommendations? 
2. How effectively do teachers interact with the TRT? 


Data Collection 


Context 

The data for this study was collected in from high school geometry students and their teachers. The 
demographics of the school, located in a rural mid-Atlantic region of the United States, were 12% Black, 44% 
Hispanic, and 38% White students, with 68% of the students receiving free or reduced lunch and 39% of the 
students classified as having Limited English Proficiency. The students participated in a mathematical modeling 
project that focused on how scale factor impacts the dimensions, volume, and surface area of a rectangular prism. 
During the project, the students were asked to write explanations for three different question prompts. The student 
explanations and teacher responses from a prior implementation of the project were used to create the training 
datasets used to answer research question 1. The teacher interactions with the TRT during a subsequent 
implementation of the same project but with different students were used to answer research question 2. 


Research question 1 

In order to assess the effectiveness of the TRT in terms of how accurately it selects recommendations we 
created three training datasets as describe above, with a different dataset for each of three question prompts (see 
Table 2). High school geometry students 


Table 2.4 summary of the three training datasets used to assess the accuracy of the TRT at selecting 


recommendations. 
Training Question Number of student explanation Number of different 
dataset prompt and teacher response pairs teacher responses 
How does the scale factor change 
1 the length, width and height of the 116 29 


object? 

How does the scale factor change 
the volume of the object? 

3 How does the scale factor change 85 6 


the surface area of the object? 


2 99 6 


For each training dataset we performed a leave-one-out cross-validation (Borra & Di Ciaccio, 2010). To do 
this for a given training dataset we first removed one of the student explanations and its corresponding teacher 
feedback. Second, we used the remaining training data to build a tf-idf model as described above. Third, we used the 
removed student explanation to query the model, and finally, recorded whether the recommendations obtained from 
the query included the teacher feedback corresponding to the removed student explanation. If so, we counted this as 
a success; if not, a failure. This process was repeated, leaving out a different student explanation from the training 
dataset each time, until all the student explanations in the dataset had had their turn to be left out. The proportion of 
successes for each training dataset was found by dividing the total number of successes by the size of the dataset, 
and to account for successes that are expected by random chance we calculated kappa for each dataset. A kappa of 0 
indicates that all the success is due to randomness and a kappa of | indicates success every time (Cohen, 1968). 


Research question 2 

To assess the effectiveness of the TRT in terms of how teachers interact with the user interface, four 
classroom teachers were observed using the user interface while following a think-aloud protocol which encouraged 
the teachers to verbalized their thinking as they interacted with the tool. Video screen capture, audio, and researcher 
fieldnotes were recorded throughout. In addition, the four teachers also participated in individual interviews several 
days later with included questions about their use of the TRT. The transcripts of think-aloud and post-project 
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interviews were analyzed to identify reoccurring themes and evidence that confirmed or diverged from the themes. 
The screen capture video was analyzed for the frequency that recommendations were selected or edited by the 
teachers and the time that they spent interacting with the TRT. 


Results 
System Effectiveness 


Research question 1 

After performing leave-one-out cross-validation for each of our training datasets we found kappa values of 
0.51, 0.84 and 0.76 (see Table 3). These results are comparable to those found in other studies that used natural 
language processing technology with open-response items. Liu and colleagues (2014) report average kappa values 
for such studies to be between 0.62 and 0.81. However, the questions considered in these studies only distinguish 
between two and five categories of response, making them more comparable to the question 2 and question 3 dataset 
results rather than the result for the question | dataset which is lower, we expect, due to the high number of unique 
recommendations included in this dataset. Therefore, we conclude that the accuracy of the TRT recommendation 
selection is comparable to those used by other studies. 


Table 3. Number of explanations, unique recommendations, proportion of successful recommendation selections, 
and kappa values by question 


Number of student Number of Proportion of 
Training dataset explanations unique recommendations successful selections Kappa 
Question 1 116 29 0.560 0.509 
Question 2 99 6 0.919 0.838 
Question 3 85 6 0.882 0.764 


Research question 2 

Teachers interacted intuitively with the TRT interface. None of the teachers were observed asking about 
how to use the interface or expressing frustrations with the interface while responding. Instead, teachers were 
positive about their interactions, for example, Henry commented that “it was a very clever interface” and that “it was 
nice to be able to see what [the students] did, try to give a tailored response to give them a hint towards where they 
supposed to be going, and it was also nice to be able to personalize it for them.” 

Teachers interacted thoughtfully with the TRT interface. The teachers interacted with the recommendations 
provided by thoughtfully considering the merits of the different recommendations with respect to the student 
explanation. Mike described that he would “look at the recommendations and think ‘Well, that one clearly isn't what 
I see happening here. This one is the closest to [the student explanation], but I think I need to just qualify it a little 
bit, modify it to fit this situation.”” And Sam said that the recommendations were “something to start off of and 
decide if I agreed with what was there, or if I needed to make up my own.” Nina commented, it was beneficial to her 
that the recommendations were not “everything I want to say as verbatim exactly what I want ... because if it was 
exactly like what I wanted to say, then I feel like [responding to students] would just be a little more mindless for 
me.” 

Thoughtful teacher interactions were supported by the functionality of the TRT interface. The teachers 
thought that the TRT selected the recommendations well. For example, Henry commented that the TRT “generally, 
did a good job pulling recommendations that fit the situation. Many of them I was able to use.” However, because 
the recommendations were often not exactly how a teacher wished to respond to a student, the teachers made use of 
the user interface functionality for selecting and editing the recommendations. As Nina explained, “I could kind of 
pick apart different pieces. It was more of editing, manipulating, or rephrasing what was already given.” The results 
from the analysis of the user interaction data collected from the screen capture video confirm that teachers interacted 
with the recommendations often while responding to students. On average across all teachers, one fourth of the 
teacher responses were unedited recommendations and half of the responses were edited recommendations. At the 
same time, there was some divergence in how the teachers used the recommendations, with Sam mostly writing 
responses without using the recommendations, and Mike mostly using unedited recommendations (see Table 4). 
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Table 4. Average responding time (in seconds) and the use of recommendations in responding, by teacher. 


Number (percentage) of submitted teacher responses that used: 


Average responding no an edited an unedited 
Teacher time (seconds) recommendation recommendation recommendation 
Sam 76.7 5 (56%) 3 (33%) 1(11%) 
Nina 69.6 4 (14%) 21 (75%) 3 (11%) 
Mike 48.2 4 (14%) 7 (25%) 17 (61%) 
Henry 84.7 5 (29%) 10 (59%) 2 (12%) 
Total 66.2 18 (22%) 41 (50%) 23 (28%) 
Discussion 


The Teacher Responding Tool (TRT) was designed and developed to scaffold teachers’ skills at giving high 
quality, student specific feedback. This required that the text of the recommendations aligned with research based- 
practices, that appropriate recommendations were selected for a given student explanation, and that teachers were 
able to interact thoughtfully with the selected recommendations via the user interface. The results from this study 
demonstrate that, for the context in which this pilot version of the TRT was tested, i.e. in the context of high school 
students writing explanations of their understanding during a mathematical modeling project, these requirements 
were satisfied. The TRT was able to select recommendations as accurately as other natural language processing tools 
and, importantly, the teacher users considered the recommendations selected to be appropriate. The user interface 
design supported thoughtful teacher interactions by providing three recommendations, functionality for selecting and 
editing recommendations, and a low extraneous cognitive load layout. A prior study (Bywater, Chiu, Hong, & 
Sankaranarayanan, 2019) demonstrated that thoughtful teacher interactions with the recommendations contributed to 
improved teacher responding practice. This study provides evidence that these interactions were facilitated by the 
TRT design. 


Recommendations 


Several recommendations for designing natural language tools for learning follow from the results of this study. 

1) Natural language processing (NLP) tools that are designed for learning should understand their impact on 
teacher professional skills. In learning contexts, NLP tools are typically used to automatically respond to 
students so that teachers are able to focus their time on those students who are most in need (e.g. Gerard, 
Matuk, McElhaney, & Linn, 2015). In this study, the TRT design included teachers into the responding 
process so that teachers had opportunities to notice how their students were thinking and to develop their 
responding skills. We recommend that designers of NLP tools for learning consider how to they can 
support research-based teacher practices. 

2) Training the system requires specific and purposeful data. The requirement of ‘big data’ is often associated 
with natural language applications and might be thought to limit the applicability of NLP techniques to 
specific learning contexts. This study suggests otherwise. Training a system with a smaller dataset that is 
specific to the applied context can be effective and can support the use in specialized, non-normative, or 
underrepresented learning contexts. 

3) Teacher input into the training dataset content is critical to generate authentic, rich recommendations. The 
process of creating the training dataset might also be considered a novel professional development activity 
that builds upon established practices within the field for teachers collaboratively examining student work. 

4) For tasks that are cognitively demanding, extraneous cognitive load can be reduced by using a minimalist 
user interface design that retains all necessary functionality. In this study, reducing extraneous cognitive 
load involved both how to best present information and how to reduce the load associated with interacting 
with the information. The TRT user interface combined the select-copy-select-paste steps that a user 
frequently repeats into a single click. This simplified the interaction steps for teachers when selecting and 
editing. 

5) Connecting the TRT with different learning management systems requires permissions to share identifiable 
user data and technical expertise. These challenges continue to present adoption hurdles but are being 
addressed within the educational technology community (e.g. Learning Tools Interoperability, 2019) and 
we recommend common standards and protocols to mitigate these challenges. 
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Abstract 


Text-based discussions have been used in online discussions arguably since the inception of online 
education. However, in recent years, different platforms have been introduced to enable video-based discussion. 
This study aims to find if there are differences between the success rate of the students who use text-based 
discussions and video-based discussions. The result of this study will contribute to the value of each discussion 
format, and discuss possible recommendations to facilitate the effectiveness and engagement. 


Introduction 


A major component of any online course is how one, often the instructor, creates a sense of community. 
Having discussions, has been one way to achieve this, which has led to online discussions becoming a key feature in 
asynchronous learning activities. The topic on online discussions has been explored in various ways; for example, 
Russo and Benson (2005) discuss the importance and relationship between online presence and cognitive learning 
outcomes. Their study focused on students and instructors’ perceptions of how online presence affects learning. 
Other studies have shown that text-based discussions have been used widely for a long time as a method to show 
students’ presence in an online environment. Video-based discussions also have been proved as a rich and powerful 
model to foster online learning (Yousef, Chatti, & Schroeder, 2014). 

There are numerous theories that can be used to explore this topic. One such theoretical framework is 
Community of inquiry (Col) proposed by Garrison, Anderson, and Archer (2000) includes three independent 
elements: social presence, teaching presence, and cognitive presence in online learning. Cognitive presence plays an 
important role that allows any participant in a community to produce productive communication, especially in 
online education. Teaching presence emphasizes the role of instructors in designing, developing, and delivering 
online learning activities. Social presence supports cognitive presence by focusing on the role of participants in an 
online community. These three basic elements have been applied in research studies to create interactive and 
meaningful online learning environment. 

A study by Clark, Strudler, and Grove (2015) investigates the differences in social presence and teaching 
presence when students communicate via video versus text-based discussions. This study captures the value of using 
video and suggest the combination multi-models in online discussions. According to Johnson and Lock (2019), 
online learning using text and video discussions will foster higher-order thinking in enhancing interaction and 
engagement. 

It is clear that most of the previous studies focus on comparing text-based and video-based discussions in 
online learning (Clark, Strudler, & Grove, 2015; Swartzwelder, Murphy, & Murphy, 2018) or investigate the social 
presence or students engagement. There is not much information about the relationship between students’ success 
and video-based and text-based discussions. According to Greogry (2016), a number of factors have been used to 
describe successful online learners, such as GPA, goal-oriented, organized, responsible, self-disciplined, and self- 
motivated. 
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Based on the above characteristics of successful online learners, this study will focus on investigating how 
text-based and video-based discussion relate to students’ success in online learning. The study is guided by the 
following research question “Is there any difference between the success rate of the students who use text-based 
discussions and those who use video-based discussions?” 


Methods 


This study focuses on finding graduate students’ success in online discussion. For that purpose, the 
researchers chose the survey method, which provides descriptions of participants’ trends, attitudes, or opinions of a 
group sample (Creswell & Creswell, 2014), to collect data for this study. 

The researchers sent emails to recruit graduate students at Oklahoma State University, who have taken 
online courses. The purposes of choosing these participants were (1) all participants were involved directly in online 
discussion, and (2) they had experiences about text-based discussions and video-based discussions in online 
learning. These selections of participants served as important factors to provide the strongest data for this study. 

After obtaining IRB approval, the researchers created an online survey, including both open-ended and 
closed-ended questions in Qualtrics. An anonymous link was then generated and distributed via emails. Participants 
(N=66) provided answers to the questionnaires. 


Findings 


This section provided main findings from the data collection to answer the research question “Is there any 
difference between the success rate of the students who use text-based discussions and those who use video-based 
discussions?” The data showed that more than 80% of the participants had taken an online course in the last two 
years; thus, the participants were able to express their experiences of online discussions. 

Table 1| illustrated the mode of discussions in online learning and graduate students’ preference. As can be 
seen from the table, text based discussion was chosen as the most preferred mode for online discussions. 


Table 1. Participants’ Preference of Discussion Mode 


Text Based Discussions (i.e. discussion boards) 46.96 
Video Based Discussions (e.g. FlipGrid) 19.70 


I have no preference 16.67 


Another question asked participants about their success in an online class based on Gregory’s (2006) 
definition of students’ success in online learning. All participants indicated that they were successful in an online 
class (Table 2). From this response, it can be confirmed that the success of graduate students in an online course was 
not affected by a specific discussion mode. 


Table 2. Participants’ Success in an Online class 


Have you been successful in an online class? 
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Discussion 


The results of this study indicated that graduate students were successful in an online course as well as in 
online discussions. It can be inferred from the finding that there is no major difference between students’ success 
and modes of online discussions. Both text-based or video-based discussions can be a great platform for online 
discussions. The results also showed that most of the participants prefer using texts when participating in online 
discussion boards. These results were consistent with participants’ cognitive presence in online learning as described 
by Garrison, Anderson, and Archer (2000). Based on the results, online instructors should consider applying 
methodologies and implementing different discussion modes in online courses. 

Because this study collected data from a small sample size, N = 66, and only focused on online graduate 
students, it is difficult to generate the results to different online learners, such as undergraduate students. Another 
limitation of this study was that the study analyzed data from graduate students’ awareness of their success in an 
online course. Further research can collect data from a larger sample size to confirm the relationship between 
different modes of online discussions and learners’ success. Additional data should be collected to examine 
students’ perception of text based discussions and video based discussions. The role of instructors and students in 
online discussions should be considered in order to examine graduate students’ success in online learning. 
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Abstract 


As distance education programs become more common and attract more students, institutions are grappling 
with ways to support online students at a distance. Graduate students face numerous obstacles creating meaningful 
academic and professional relationships that on-campus graduate students do not encounter. Supported by statistics 
and literature, this paper illustrates how students and a professor in the Instructional Systems Design and 
Technology doctoral program at Sam Houston State University utilize social media as a communication and 
collaboration tool, to facilitate the minimization of these obstacles by creating socially interactive and supportive 
peer-peer and instructor-peer communities. 

Technology, coupled with the Internet, has recreated the landscape of much of our day-to-day lives. The 
advent of social media has simultaneously made the world larger and smaller, and this has significant implications 
for postsecondary distance education. Since social media has gained prominence, educators and researchers have 
explored the applicability of this tool in a learning context. Building connections and relationships between 
professors, students, and peers emerge as crucial components of online learning environments, which are on the rise 
(Babson Survey Research Group, e-Literate, & WCET, 2017; Haythornwaite, Kazmer, Robins, & Shoemaker, 
2018). These relationships are especially important for graduate students who are preparing for academia and 
professions (Gersick, Bartunek, & Dutton, 2000). 

Further, social media has made it easier to access and create networks by providing easier ways to connect 
with increasingly large amounts of people and information. Watts and Strogatz (1998) introduced the idea of 
““small-world’ networks ... popularly known as six degrees of separation” (p. 440). This concept showcases the 
strength of a dynamic network consisting of nodes in which ideas, information, and disease can spread quickly via 
shortcuts. Inspired by this idea, this paper aims to examine how social media networks cross-connect with distance 
education, various learning theories such as social constructivism and transactional distance, and recommended 
strategies. 
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Status Update: The State of Distance Education, Social Media, and Theory 


Distance education and face-to-face networking have existed for a very long time, and there is no doubt 
that time and technology have transformed what this looks like today. While social media is relatively nascent, it, 
too, has undergone tremendous growth and transformation in its short life span. The connection between the three 
has created a modern learning context in which the lines between work, school, and private life often blur. 


Distance Education 

Distance education has its roots in correspondence studies of the late 1700s (Keegan, 2002). However, the 
modern definition has been redefined as a system of “teaching and planned learning in which teaching normally 
occurs in a different place from learning, requiring communication through technologies as well as special 
instructional organization” (Moore & Kearsley, 2012, p. 2). As the delivery method has moved from the postal 
service to the modern era of web-connected devices, the distance learner has come to include a broader range of 
people. 

As of 2015, almost 6 million students in higher education were participating in at least one distance 
education course (Babson Survey Research Group, e-Literate, & WCET, 2017). These figures are estimated to 
contribute to almost 32% of higher education enrollments as of 2016 (Seaman, Allen, & Seaman, 2018). 
Interestingly, while overall enrollments in traditional higher education programs are decreasing, enrollment in 
distance education programs is increasing, as shown in Figure | (Babson Survey Research Group et al., 2017). 
Generally, public institutions are outpacing private schools in offering distance instruction, as more students take 
advantage of the available opportunities (Allen, Seaman, Poulin, & Straut, 2016). 
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Figure 1. Overall Trends in Distance Education in The United States 


Adapted from “Grade increase: Tracking Distance education in the United States,” by J. Seaman, I. Allen, and J. 
Seaman, 2018, http://onlinelearningsurvey.com/reports/gradeincrease. pdf. 

The overall trend towards increased participation in distance education also holds true for both 
undergraduate and graduate programs, as shown in Figure 2 (Seaman et al., 2018). As a result, educational leaders 
are taking note and consider online education options as part of their long-term institutional viability and strategy 
(Allen et al., 2016). While at times, there have been varying levels of confidence regarding the quality of distance 
education programs, this perception is quickly changing, and the majority of education leaders believe the quality is 
on par or superior to traditional face-to-face learning (Allen et al., 2016). 
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Figure 2. Comparison of Overall Distance Education Undergraduate and Graduate Enrollments in the United States 


Adapted from “Grade increase: Tracking Distance education in the United States,” by J. Seaman, I. Allen, and J. 
Seaman, 2018, http://onlinelearningsurvey.com/reports/gradeincrease.pdf. 


Demographics and Characteristics of Distance Education Students 

In higher education settings, almost 30% of students participate in distance education courses, with nearly 
half exclusively enrolled in distance coursework (Babson Survey Research Group et al., 2017). The majority of 
these learners attend public universities, followed by non-profit private and for-profit private institutions, 
respectively (Babson Survey Research Group et al., 2017). Additionally, the majority of distance education students 
are enrolled in home-state institutions (Seaman et al., 2018). While there are differences in the number of graduate 
and undergraduate students enrolled in distance courses depending on the type of institution, graduate students are 
still more likely to pursue distance coursework compared to undergraduate students (Babson Survey Research 
Group et al., 2017). Of these students, the majority considered to be exclusively enrolled in distance programs tend 
to live in the same state as the institution they attend (Babson Survey Research Group et al., 2017). 

Some research indicates that the demographics of distance learners is characteristic of the U.S. as a whole 
and that most are looking to advance their career (Cleary-Estep, 2016). Others find that distance learner 
characteristics are fluid and constantly in flux (Latanich, Hudson Gail, & Nonis Sarath, 2001). Still, distance 
learners generally tend to be older than traditional students (between the ages of 25-50), female, and work full-time 
(Cleary-Estep, 2016; Halsne & Gatta, 2002; Latanich et al., 2001; Moore & Kearsley, 2012; Smith, 2014). The 
specific percentages in these categories may vary, but the inclination remains valid. Distance learners also have a 
propensity to make more than a $40,000 salary and are predominantly white (Halsne & Gatta, 2002; Shriner, 2015). 
However, African-American and Hispanic student enrollments are significantly higher in distance education 
programs compared to that of traditional environments (Shriner, 2015). While there are some distinguishing 
characteristics, one study reports that online and traditional students share similar traits in the areas of marital status, 
socio-economic status, level of education, and level of employment (Pentina & Neeley, 2007). 

Just as there is variation in demographics, distance education programs seem to attract students with certain 
characteristics. The convenience of distance education due to its flexibility may no longer be novel and requisite, as 
students look for quality, personalization, individualization, and convenience (Pentina & Neeley, 2007). Overall, 
learners who choose distance education programs display higher levels of motivation and risk-taking (Latanich et 
al., 2001). Coupled with more experience in higher education and full-time employment (Latanich et al., 2001), 
instructors have an opportunity to leverage these characteristics when designing distance learning experiences. 


Quality of Effective Distance Education 

The quality of distance education programs has been a long-standing topic of discussion. Although the 
tools to deliver distance education have changed, Keegan (2002) notes that “distance education has a history of more 
than 150 years, where institutions [have] offered high-quality education to learners ‘free of time and place’” (p. 92). 
Rather than focusing on whether to offer distance opportunities, institutions should focus on quality assurance 
measures that evaluate the effectiveness of distance learning and accreditation (Carlsen, Holmberg, Neghina, & 
Owusu-Boampong, 2016; Muilenburg & Berge, 2001). 
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Once course quality is present, attention must turn to learner needs and perceptions. According to Lawless 
and Richardson (2002), students’ perceptions of distance education are associated with their approach to studying; 
this implies that course design should address effective study skills specifically for distance learning (Lawless & 
Richardson, 2002). This opinion is also informed by the judgment of workload appropriateness, clarity of goals, 
assessment, materials, and teaching quality (Lawless & Richardson, 2002). Instructors must be able to provide clear 
course objectives, organizational structure to the course, clarity of instructions, and a choice between group and 
individual projects (Jones & Blankenship, 2017). Moore’s theory of transactional distance supports these assertions 
on distance learning (as cited in Vasiloudis, Koutsouba, & Giossos, 2015). 

Faculty and instructional designers must also be prepared to construct quality instruction specifically 
targeted for virtual distance environments. Institutions must ensure ongoing professional learning is provided to 
faculty on pedagogy, technology integration, and appropriate instructional strategies for distance learning (Broady- 
Ortmann, 2002; Falowo, 2007). As discussed previously, maintaining the quality of communication and social 
interactions in an online environment is essential for distance learning success (Copeland & Warren, 2004; 
Haythornthwaite, Kazmer, Robins, & Shoemaker, 2018; Mayisela, 2013). 

Universities will also have to reevaluate pedagogical considerations for distance learning, adult learners, 
and collaboration in an online environment (Gearhart, 2001). Pedagogy in online education environments must 
address intellectual engagement, communication, collaboration, and discussion (Lee, 2017; Xin & Feenberg, 2006). 
The key to success for online education appears to be increased interaction with the instructor and content (Mbwesa, 
2014). As Ascough (2002) stated, “good pedagogy requires an awareness of the opportunities and limitations of the 
mode of education” (p. 1). Institutions should also ensure that online programs meet standards of rigor and quality; 
they must also address the unique aspects of distance education (Stella & Gnanam, 2004). As a result, online courses 
must provide students with ongoing social interactions that elicit analytical and reflective discussion with feedback 
(Jones & Blankenship, 2017). 


Table 1. Top Social Media Platforms Amongst Global Internet Users, Excluding China 
Platform Founded | Description Estimated Users 
Facebook 2004 Platform to connect with friends 2.2 billion 
YouTube 2005 Platform for video sharing 1.8 billion 
WhatsApp 2009 Smartphone messaging application 1.3 billion 
Facebook Messenger | 2011* Messaging application 8 million 
Instagram 2010 Platform for sharing images and videos 1 billion 
Twitter 2006 Microblogging platform 1.6 billion*** 
Google+** 2011 Platform to connect with friends 351 thousand*** 
LinkedIn 2002 Professional networking site 575 million**** 
Skype 2003 Video and voice call platform 23.6 million*** 
Snapchat 2011 Messaging app for time-limited images and videos 186 million 


*Based on Facebook Chat released in 2008. 
**Shut down for users with personal accounts in April 2019 (“Consumer Google+ shutdown,” 2019). 


*** Organic users. 


**** Registered users. 


Adapted from “The history of social media: Social networking evolution!” by K. Terrell, June 16, 2015, History 
Cooperative, https://historycooperative.org/the-history-of-social-media. 
Adapted from “15 most popular social media networks and sites [+160 data points],” by C. Brenner, July 18, 2018, 
G2, https://learn.g2.com/social-media. 


Social Media 


In order to create effective and engaging online instruction, the rise and impact of social media in higher 
education needs to be explored. Boyd and Ellison (2007) use the term social network site to describe a web-based 


service in which users create a profile that allows them to connect with other users and their respective connections 
in a manner that elevates the relationship. Furthermore, these connections among users are transparent to everyone 
on the platform (Boyd & Ellison, 2007). As the terminology has shifted towards the use of “social media,” these 
platforms are now web-based and mobile apps in which users can connect and communicate through user-generated 
content (Kaur Kapoor et al., 2018). 

The emergence of social media platforms can be traced back to 1996, but truly emerged in their modern 
form in the early 2000s (Singh, 2019). With platforms such as LinkedIn, Facebook, Twitter, Instagram, YouTube, 


60 


and more, the number of worldwide users has jumped from almost a billion in 2010 to 2.62 billion in 2018 
(McFadden, 2018; Singh, 2019). Messaging platforms such as Facebook, Snapchat, WhatsApp, and Facebook 
Messenger tend to be the most popular platforms, as shown in Table 1 (GlobalWebIndex, 2018). 


Demographics in Social Media 

Focusing on users in the United States, the Pew Research Center has been tracking social media trends 
since 2005 (Perrin, 2015). Amongst adults, social media use has skyrocketed from 7% in 2005 to 65% in 2015 
(Perrin, 2015). Globally, 98% of Internet users are also social media users who have an average of 8.5 accounts 
(GlobalWebIndex, 2018). Generally, social media platforms tend to be used for different purposes 
(GlobalWebIndex, 2018). While 18-29-year-old young adults have the highest rate of usage at 90%, all demographic 
age groups have shown increases in adoption rates (Perrin, 2015). There are similar usage rates across gender, race, 
and ethnicity; however, social media is used at higher rates by those living in urban areas and attaining higher levels 
of education and income, as shown in Figure 3 (Perrin, 2015). 
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Figure 3. Demographic Trends for Social Media Users in the United States 


Adapted from “Use social networking sites report (Pew Research Center Report),” by A. Perrin, 2015, 
https://www.pewinternet.org/2015/10/08/social-networking-usage-2005-2015. 


Social Media Use in Higher Education 

Most data on social media usage in higher education tends to focus on younger undergraduate students, 
which is not representative of non-traditional distance education students, as previously outlined. However, this data 
is still instructive in developing an understanding of use in higher education settings. On-campus undergraduate 
students report that they prefer platforms that focus on personal relationships rather than professional networking 
(Knight-McCord et al., 2016). As a result, students preferred using Instagram, Snapchat, Facebook, and Twitter, 
respectively (Knight-McCord et al., 2016). Overall, undergraduate students tend to maintain a presence on multiple 
platforms, feel that it can improve academic discussions, and is more convenient (Jacquemin, Smelser, & Bernot, 
2014). 

Graduate student use of social media illustrates a shift towards incorporating professional and academic 
pursuits into their social media use. Much like undergraduate students, graduate students show a preference for 
Google+ (no longer available to the public), Facebook, Twitter, and Instagram, respectively (Romero-Hall, 2017). 


61 


While they use these platforms primarily for personal use, there is increased usage of platforms such as LinkedIn for 
connecting to academic or professional communities (Romero-Hall, 2017). Graduate students tend to use fewer 
platforms and feel that it is far more convenient than using other online options such as those provided within an 
LMS (Jacquemin et al., 2014). 


Instructional Uses 

Instructionally, social media platforms are being incorporated into many aspects of higher education. The 
Internet and social media have created a vast depository of knowledge that is decentralized and always available to 
both instructors and students (Cortés & Lozano, 2014). In one study, graduate students felt that using Twitter 
exposed them to information they may not have otherwise found (Jacquemin et al., 2014). While students sometimes 
feel uneasiness in using social media as a learning tool, they often find that their opinion changes as they see the 
utility of transparent communication with professors and other classmates (Bista, 2015). In addition, many students 
also found value for social media use in their professional or personal applications (Bista, 2015). Other studies 
highlight the varied use of Twitter for instruction, faculty communication, and professional development (Lewis & 
Rush, 2013). 


Communities of Learning 

Oftentimes, social media can also be used to develop informal communities of learning. In these cases, 
people with similar interests can develop or find communities with which to connect and share information (Lewis 
& Rush, 2013). While common interest communities and collaboration naturally occur in many face-to-face 
environments, social media has the power to enlarge the circle of people with whom you interact (Amin & 
Rajadurai, 2018). Moreover, social media affords the prospect for dialog between communities (Kumar & Nanda, 
2019). According to Lewis and Rush (2013), social media plays a role in creating useful networks or communities of 
practice for those in higher education. 


Relationship Status: Social Media + Distance Education + Theory 


The use of social media within online distance education programs seems to be a natural fit; multiple 
educational theories support this conviction. Next, the relationship between social media, distance education, and 
various learning theories will be examined. 


Learning Styles 

While the term learning style generally refers to a method by which students learn best, there are many 
different ways to approach this concept. Gardner (1991) proposed multiple intelligences in which learners may have 
linguistic, logical-mathematical, spatial, musical, bodily-kinesthetic, interpersonal, and intrapersonal strengths. 
Learning styles also frequently refer to modalities, such as visual, kinesthetic, auditory, and tactile (Fleming & 
Baume, 2006; Lake, Boyd, & Boyd, 2017; Sousa, 2001). While these are two of the most common interpretations, 
learning styles may also address personality, learner preferences, and learner approaches (Lake et al., 2017). 

According to Halsne and Gatta (2002), online learners have historically tended to prefer visual instruction, 
while traditional students prefer auditory and kinesthetic instruction. Many students like the instructional use of 
social media due to its high use of audio-visual content, which is in line with this idea (Stathopoulou, Siamagka, & 
Christodoulides, 2019). Others suggest that social media’s ease of use may be more important than learning style 
(Balakrishnan & Gan, 2016). While using multimedia for instructional purposes is not new in distance education, 
social media platforms provide the added opportunity to design “rich learning situations” enhanced by multimedia 
and social networks (Cortés & Lozano, 2014, p. 67; Reiser, 2018). 

Instructors must also take into account the skills a student must acquire in order to achieve the intended 
learning outcome (Roblyer, 2015). Gagné (1985) proposed five categories of learning: verbal information, 
intellectual skills, cognitive strategies, attitudes, and motor skills. Similarly, Bloom classified skills into cognitive, 
affective, and psychomotor domains (Anderson et al., 2001; Roblyer, 2015). In this context, the instructional 
objectives determine the skills a student is expected to acquire, and the instructional design should reflect 
appropriate sequence, strategies, and media to achieve the specified outcome (Roblyer, 2015). Because there are 
such a variety of social media platforms that can be used in many different ways, these platforms can almost always 
meet some preferences or needs of the learner (Stathopoulou et al., 2019). 
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Constructivism 

Constructivism serves as an overarching paradigm in which many individuals have contributed significant 
theories of how learners construct knowledge. According to Piaget, learners create knowledge by interacting with 
others through a social and physical environment (Schrader, 2016). While Piaget focused on a biological perspective 
of cognition, Lev Vygotsky proposed a sociocultural theory in which cognitive development is built from the social 
and cultural connections of the learner (Woolfolk, 2001). In this sense, learning serves an interdependent social 
function before it is internalized into individualized knowledge (“Social development theory (Vygotsky), 2014). 
Constructivism promotes two essential ideas. The first is learners should be provided a learning task that is within 
their cognitive ability level, and second is students can achieve beyond their independent level with support 
(Woolfolk, 2001). In the context of social media, these platforms provide ample social interaction opportunities with 
people of all levels. There is initial evidence that using social media with a constructivist pedagogy can lead to 
improved academic performance (Amin & Rajadurai, 2018; Hashim, Rashid, & Atalla, 2018). According to Amin 
and Rajadurai (2018), social media platforms lend themselves to the aspects of the ideal constructivist classroom. 

The reality is that online learning experience is based in social and cognitive development. According to 
the Community of Inquiry Framework, online education is based on the dynamic relationship between social, 
cognitive, and instructor presence (Garrison, Anderson, & Archer, 2010). Social and cognitive engagement is also 
integral to the success of the distance learner. In fact, instructors who create a digital culture of dialogue, risk-taking, 
and interaction between students and the instructor increase the feeling of social inclusion for distance learners, and 
by extension, the potential for success (Brown et al., 2012). 

Cognitive presence integrates Dewey’s ideas on reflective inquiry, while social presence relies on building 
community through purposeful communication (Garrison et al., 2010). The instructor’s presence may be a key 
element in which an instructor establishes structures and processes that facilitate learning through a community in a 
manner that addresses the cognitive and affective domains (Garrison et al., 2010; Roblyer, 2015). 

Of course, these are not the only theories that can help explain why social media has the potential to be an 
ideal instructional tool or environment. Situated cognitive theories lend credence to the idea that knowledge has 
meaning within authentic contexts, while connectivism promotes having leamers link ideas, information, and people 
(Cortés & Lozano, 2014; Schrader, 2016). 


Transactional Distance 

Moore’s theory of transactional distance posits that distance can be a geographical and pedagogical 
experience (Moore & Kearsley, 2012). There are five types of interaction in which transactional distance can be 
measured: learner-instructor, learner-learner, learner-content, learner-interface, and learner-environment (as cited in 
Yilmaz, 2017). Yilmaz (2017) found that the Facebook environment produced positive perceptions of transactional 
distance. Social media studies have supported learners increased interaction in all transactional distance 
relationships (Amin & Rajadurai, 2018; Bista, 2015; Lewis & Rush, 2013). Studies have found that social media use 
for learning included increased use of resources, quality interactions, engagement, interest, and collaboration (Ricoy 
& Feliz, 2016; Tess, 2013). 
Connectivism 

The idea that social media increases connectedness and interactions among distance education 
environments is becoming more visible; this has led to new theories about how we might learn best in this new 
digital age. Connectivism is a theory proposed by George Siemens and Stephen Downs, which suggests that the age 
of ubiquitous technology and connections has changed the way people learn (Ungvarsky, 2019). Whereas 
knowledge used to be stored in textbooks and disseminated by the instructor expert, learners now continually access 
and assess their learning via networks, which is markedly different from the linear path instruction used to follow 
(Ungvarsky, 2019). In other words, students must be active participants in co-creating their own learning experience 
(Cortés & Lozano, 2014). 


Direct Message: The Future of Social Media and Distance Learning 


As exclusively online students in a graduate program, the research presented in this paper resonates with 
the writers’ experiences. In what is often an isolating, lonely, and arduous path of independent learning and research, 
social media has provided a user-friendly platform in which to connect and learn from peers in this program. It has 
afforded the writers much-needed forums to clarify, mentor, educate, ask questions, celebrate successes, share and 
learn from failures, complain, and seek or provide motivation. Social media has provided opportunities to connect 
and learn from experts in the field that would have otherwise been unavailable in a traditional face-to-face program, 
which has resulted in tremendous growth of professional networking opportunities. While there is no doubt that 
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social media has allowed a deeper level of learning and application, the overarching benefit is the lifelong 
relationships that have developed and grown into friendships based on respect, collaboration, and peer mentoring. 
Social media is undoubtedly impacting and changing education in ways that will surely be more visible or 
measurable in the future, yet in its relatively short existence, it has already changed how the authors learned and 
view learning. Discussing higher education strategies for graduate students can be categorized as institutional, 
instructional, and professional. 


Institutional Strategies for Online Graduate Students 

Institutions must always determine how to enhance the student experience in order to retain students 
(Aversa & MacCall, 2013; Britto & Rush, 2013). When discussing online versus on-campus programs, many point 
to the difference in services and experience as a key decision-making factor in the selection process (Nelson, 2017). 
According to Britto and Rush (2013), the challenge has been in simply offering online students comparable services 
already provided to face-to-face students. These include services such as online technical support, safety alert 
systems, online advising, support for first-time, fully online students, student readiness assessments, online student 
orientations, online tutoring, and school newsletters. Social media can connect students with the right services or 
communities of support designed for the distance learner. 


Instructional Strategies for Online Graduate Students 

Instruction is the most important element of any educational institution. When discussing online 
instruction, the role of the instructor is often deeply intertwined with the design and delivery of a course (Fedynich, 
Bradley, & Bradley, 2015). While feedback was generally positive for the online experience, Fedynich et al. (2015) 
noted that students reported lower levels of satisfaction with instructor feedback and peer interactions. 
Unfortunately, some instructors find online instruction and technology platforms difficult to use, which can lead to 
poor instructional experiences for students (Bawa, 2016). 

Kumar and Nanda (2019) offer many opportunities for how social networking can be used instructionally in 
higher education. These suggestions include student coursework collaborations, quality online feedback, and online 
communities to support increased student engagement, participation, and learning (Hashim et al., 2018; Kumar & 
Nanda, 2019). With the proper instructor support, learners can be empowered and gain confidence as participants 
and learners within a social media community (Ricoy & Feliz, 2016). The authors have experienced first-hand how 
social media can be used to authentically connect with, learn from, and obtain feedback from others in the field. 
These types of experiences elevate an assignment into an authentic collaboration with an artifact of learning that 
furthers a professional social media presence. 


Professional Strategies for Online Graduate Students 

Online graduate students tend to already be working professionals (Cleary-Estep, 2016). As a result, higher 
education institutions benefit from providing services that support the professional goals of their students. In this 
regard, social media can be a great tool to invite participation from online students who may not be able to attend 
events in person. Whether it is a campus tour, lecture series, campus communities, professional groups, or 
mentoring, social media can be used to share information and create interactive online experiences (Kumar & 
Nanda, 2019). Professionally, LinkedIn provides a unique platform that institutions can use for recruitment, 
fostering professional connections, and promoting alumni (Kumar & Nanda, 2019). As Nelson (2017) noted, online 
students often lack access to key networking opportunities such as internships. In addition to this, the authors have 
also noted a lack of opportunities for research and teaching that are often available to on-campus graduate students. 

While it may be impossible to offer an online program that is comparable to an on-campus program, the 
goal should be to provide equitable services and experiences using any available platform. There are several 
questions that higher education and instructors can ask themselves. 

How can social media be used to: 

¢ create communities of and for learners at the institutional, program, department, and course level? 

e provide online graduate students with opportunities for ongoing mentorship and research? 

e train online instructors in the effective use of technology tools for instruction, collaboration, and 
engagement? 
create authentic learning experiences? 
develop or support professional and alumni networking? 
provide online graduate students virtual research and teaching opportunities? 
support the transition from personal to professional social media use? 
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e accessing the most current information and emerging discussion on a subject? 
e extend beyond the “classroom” and include discussions or collaborations with others in the field? 
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Index descriptors: mixed methods, data use 
Introduction 


In an era of broad accountability and recent technological advances, more educational data is available now 
than ever before; but the potential benefits of data usage have not been realized (Murray, 2014; Schildkamp, 
Poortman, Luyten, & Ebbeler, 2016). As primary decision-makers for their schools, principals are expected to 
analyze large amounts of educational data to inform everything from allocating resources to promoting instructional 
strategies (Siemens et al., 2011). “Data use lives and dies in the principal’s office because principals are in contact 
with so many aspects of data use—their own data use, their teachers’ data use, and their district’s data use” 
(Wayman, Cho, & Johnston, 2007, p. 55). Educational leaders are investing in new data analytic tools (Murali, 
2014), but research to understand the process, context, and consequences of these efforts is limited (Coburn & 
Turner, 2011). 

This mixed method study provided an overview of the experiences of principals and the challenges they 
face in making sense of the vast amount of data available to them with the tools available to them. This study has 
responded to the continued need to understand how principals use data in their weekly practices at schools (Spillane, 
2012). The purpose of this study was to examine the experiences of principals using digital data analysis tools and 
identify tensions that exist between theory and practice. 


Theoretical Framework: Activity Theory 


Introduced into the field of human—computer interaction in the late 1980s, activity theory has been used to 
understand the role of tools in everyday life and how they shape people’s interaction with information 
(Clemmensen, Kaptelinin, & Nardi, 2016; Kuutti, 1996; Nardi, 1996). Building upon the work of other researchers 
in the field of education and HCI (Allen, Karanasios, & Slavova, 2011), activity theory can be used to frame and 
describe the mediated relationships among (a) a subject, the school leader; (b) the object, data; and (c) the school 
community. The objective of engaging in such activity is to gain insight, which has been described as a discovery 
resulting in new understanding (Dove & Jones, 2012). The ultimate outcome of analyzing data is to transform 
insight into a decision that will be implemented in order to have an impact (Mandinach, Honey, & Light 2006) and 
this outcome is what motivates the subject’s activity (Kuutti, 1996). The components of this framework, visualized 
in Figure 1, were selected in alignment with the literature reviewed around motivation for data analysis, factors 
influencing data use, and the intended outcome of data visualization tools. 
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Figure 1. Principals’ data use as an activity system. 
Research Methods 


Activity theory served as the foundation for shaping the research questions and research design. The four 
main research questions were: (1) What types of data and data analytic tools do principals use? (2) How do 
principals use data analytic tools? (3) What factors influence principals’ use of data analytic tools? (4) How do 
principals describe the impact of data analytic tools on their work? There were two distinct phases in the mixed 
methods study. Figure 2 shows the explanatory sequential design of this study. During Phase | quantitative data was 
collected from 70 participants through a 15-minute online survey sent to all 256 public school principals in the 
Hawai‘i Department of Education resulting in a representation of 27% of the population. The survey instrument 
consisted of subscales from two primary sources: (a) Factors Promoting and Hindering Data-Based Decision 
Making in Schools Survey (Schildkamp et al., 2016), and (b) Teacher Data Use Survey: Administrator Version 
(Wayman, Wilkerson, Cho, Mandinach, & Supovitz 2016). Descriptive statistics were applied to the information 
gathered from the survey to identify trends and select interviewees for the qualitative data collection. Six 
respondents who indicated a willingness to participate in Phase 2 and represented demographic variation to the 
greatest extent possible were asked to participate in the think-aloud and semistructured interview. Principals in 
Phase 2 participated in the 30-minute think-aloud protocol and observation in which they demonstrated how they 
used various tools, followed by a 30-minute semistructured interview about their experiences. The data were 
triangulated and coded for emerging themes in alignment with the research questions and theoretical framework. 


Followed up 


Qualitative data 


Quantitative data Followed up 


collection: D ole collection: with: 
Online survey sent bie ste * Document analysis -Coding for themes 
to all 256 principals of academic plans -Member checks 


. bs : analysis of trends 
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Figure 2. Explanatory sequential design of this study 
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Data Analysis 


The researcher followed the major data analysis steps suggested for a mixed methods explanatory 
sequential design: (a) collect the quantitative data; (b) analyze the quantitative data; (c) design the qualitative strand 
based on the quantitative results; (d) collect qualitative data; (e) analyze qualitative data; and (f) interpret how the 
connected results answer the quantitative, qualitative, and mixed methods questions (Creswell & Plano Clark, 2011). 
During the qualitative phase, instead of waiting until the end of data collection, the researcher took an inductive 
analysis approach to reflect on the meaning of what was heard in order to develop hunches (working hypotheses) 
about what was meant throughout the study. In order to represent general trends, frequency data and descriptive 
statistics were used, including measures of central tendency, such as the mean. The measures of variability, such as 
the range and standard deviation, indicated the spread of scores (Creswell, 2008). Tables and charts were generated, 
using both SPSS and Tableau. Based on the data analysis, the researcher decided which participants to invite to 
Phase 2 and what results required further explanation (Creswell & Plano Clark, 2011). Major findings from the 
Phase | quantitative data analysis were recorded into summary statements with implications for the Phase 2 
qualitative data collection. The audio recordings from the think-aloud sessions were analyzed along with the 
information from the semistructured interviews for common patterns of actions, unique behaviors, or actions that 
seemed incongruent with the verbal descriptions. After reviewing the list of generated codes from each source, the 
researcher clustered codes into larger categories and identified emerging themes (Creswell, 2008). 


Findings 


To answer the first research question, principals were asked on the survey how frequently they used various 
types of data and various tools. As shown in Table 1, there were five types of data that at least 30% of principals 
reported using at least weekly: teacher observation data (42.8%), student attendance data (34.3%), data about best 
instructional practices (34.8%), resource management data (31.9%), and feedback from teachers (30%). Data that 
provided information about resource allocations, accountability and performance, or anecdotal feedback was 
typically used once a month or less followed by student demographic data which was used least. 

Table 1. Frequency of Use by Types of Data (N=70) 


Less than Onceortwice Weeklyor A few timesa 
Donotuse onceamonth amonth almost weekly week Std. 
Data Type =1 =2 =3 Mean deviation 


Teacher observation data 21.4% 35.7% y U 3.33 0.94 
Student attendance data 25.7% 40.0% 25.7% t 317 0.92 
Data about best practices for instruction 24.6% 36.2% 26.1% . 3.10 1.02 
Resources management data 26.1% 40.6% Oi 29.0% b 3.06 0.86 
Feedback from teachers oi 34.3% 34.3% Hi 24.3% a 5. 2.99 0.94 
Effectiveness of school programs 26.1% 43.5% 18.8% a 4, 2.87 0.95 
Other B 11.8% Hi 23.5% BE 17.6% Bi 17.6% 2.82 15) 
Teacher-generated authentic assessment data 34.3% 27.1% @ 17.1% m 10.0% 2.80 1.16 
Student growth data 35.7% 42.9% Hl 15.7% = 14% 2.74 0.83 
Data about curriculum needs 33.6% W42.9% Hi 14.3% 2.67 0.77 
Benchmark assessment data 30.0% 47.1% @ 10.0% 2.9% 2.66 0.90 
Universal screener data 35.7% 47.1% = 4.3% 5.7% 2.66 0.90 
Student language data Biss.1% 31.9% w 10.1% 2.49 0.72 
Feedback from students 3.5% 36.2% w 10.1% 2.46 0.81 
School performance data B23” 24.6% m 8.7% 2.38 0.71 
Standardized test scores Biks.cx 25.7% m 5.7% 2.37 0.59 
Perceptions of the learning environment mw 5.7% Biro.0% i 18.6% a 5.7% 2.24 0.65 
Report card data Bi 13.2% Biks.2% Bi 13.2% 4.4% 2.18 0.83 
Student socioeconomic data @ 11.4% | eg B 14.3% 1.4% 2.06 0.56 
Student ethnicity data 17.4% B32. mw 58% 2.9% 1.94 0.59 
Student gender data 26.1% Biks.1% s 2.9% 2.9% 1.83 0.62 
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As seen in Table 2, four digital tools were used by the majority of principals on a weekly basis: Google 
Suite, though not supported by the district, stood out at the top (80%), followed by Excel spreadsheets (58%), 
Infinite Campus the student information system (51.5%), and tools generated at the school level (50.8%). 


Table 2. Frequency of Use for Digital Data Tools (N=70) 


Less than once Once ortwicea Weeklyor A few timesa 
Do not use a month month almost weekly Std. 
Tool =2 =3 =4 Mean deviation 
Google Suite ; = 5.7% 11.4% 21.4% i 4.27 1.06 
Excel spreadsheets : m 15.9% 20.3% 34.8% : 3.54 1.18 
Infinite Campus : m 20.6% 20.6% 20.6% d 3.47 1.32 
School-based ; m 12.7% 14.3% 25.4% . 3.19 1.51 
eCSSS : MH 22.9% 28.6% 17.1% i 3.11 1.29 
PDE3 r m 19.4% 47.8% 20.9% : 3.01 0.95 
LDS é Wi 51.4% 28.6% 11.4% i 2.54 0.88 
ARCH i By 61.4% 30.0% 4.3% ca 2.40 0.69 
HSAP Hi 60.0% 24.3% 5.1% 2.26 0.72 
iReady Bl 62.3% = 4.3% 18.8% 13.0% - od 1.87 1.21 
Other Bi 69.2% 15.4% 7.1% : 1.85 1.41 
Renaissance Learning Bi 55.9% mM 19.1% 17.6% 74% 1.76 0.99 
AVA System 54.3% Mi 37.1% 43% 4.3% 1.59 0.77 
School View Bl 63.8% Mi 33.3% - 14% 1.4% 1.41 0.60 
Jupiter Grades - 14% 2.9% . 1.16 0.69 


In the interviews, many principals talked about the benefits of the Google Suite tools because they provided 
the flexibility to collect various kinds of data, such as perceptual data, as discussed in the following example: 

For other types of data, perspective data, like, we do use Google Survey a lot, or Google Forms rather, to 
survey. Like, for example, we just did an evaluation of our kindergarten orientation with the parents, and so that was 
through a Google Form. We’II do an interest survey. And then we also use Messenger to push out the surveys. Our 
Wellness Committee put together a Cooking Up the Rainbow Night. And so we just did the sign ups over Google 
Forms, and then were able to see, you know, who signed up. Capture that data really easily. Every year we have 
STEM Ho’ ike. For the past two years we’ve been having kids showcase what they’re learning in STEM and their 
engineering design. And so—this is just simple—and it wasn’t really doing a Likert scale or anything, but we just 
asked two simple questions: What do you want to see repeated? What you want to see improved? And just capturing 
that data there. (Principal 6) 

Other staff members often factored into these activities because they shared the ability to use Google tools 
collaboratively. The majority of digital data tools supported by the district were used for student learning and 
demographic data analysis with limited school process and perception data. No common tools formally supported by 
the district for principals integrated all of the data types. 

To answer the second research question, principals that participated in the think-aloud were asked to 
demonstrate how they used each tool. Principals used data analysis tools for various purposes including making 
comparisons, planning, and reporting. Data was used to make comparisons between time intervals, schools, different 
teachers, specific groups of students, and between programs. Trends emerged showing that were used for academic 
planning, goal setting, budgeting, staffing decisions, and to make decisions about teacher support needs. 

To answer the third research question, principals answered interview questions and survey questions in 
alignment with the following subscales: (a) data use for school development, (b) school organization characteristics 
(vision and norms, leadership, and support), (c) user characteristics (knowledge and skills, dispositions to use data), 
(d) computer system characteristics (e) data characteristics (accessibility of timely data, usability, and quality) 
(Schildkamp et al., 2016). Factors that negatively influenced principal use related to the tools included problems 
with access (i.e. log in issues, privacy limits, etc.), lack of integration between systems, lack of desired functionality 
(i.e. visualizations and dashboards), and difficulty with user interfaces. Data factors that negatively influenced use 
included poor data quality (i.e. lagging, irrelevant, and unreliable data) as well as the overwhelming quantity to sift 
through. 

The nine items from the Data Use for School Development scale in Schildkamp’s Factors Affecting Data 
Use Survey (personal communication, October 26, 2017) breaks down the ways that principals might use data in the 
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context of school improvement. Principals responded to each statement on a Likert scale that included 1| (strongly 
disagree), 2 (disagree), 3 (neutral), 4 (agree), and 5 (strongly agree). The results appear in Table 3. More than 90% 
of principals agreed or strongly agreed that student results were used to determine yearly goals for school 
improvement (97.1%) and that student results led to decisions about professional development (95.6%). More than 
80% indicated that (a) they used data to show teachers the extent to which the school achieved goals (85.5%) and (b) 
student achievement results were used to identify curricular gaps (84%). Three fourths or more of the principals 
replied that detailed data analyses were essential in the improvement process (79.7%) and that data were used to 
determine effective teaching (75%); about two thirds indicated using external evaluation for improvement (66.6%). 
In two areas at least one third of principals were neutral or disagreed regarding data use: using external evaluations 
for improvement (33.3%) and division of teaching time is based on learning needs (38.2%); furthermore, nearly 
three fourths were neutral or negative in the use of student results to evaluate teachers (72.4%). The Data Use for 
School Development scale provided some insight into the way principals made use of data, but the majority of 
information collected to answer RQ2 came from the Phase 2 qualitative data collection. 


Table 3. Data Use for School Development Scale (N=70) 


Strongly Disagree Disagree Strongly Agree Std. 
=1 =2 Mean = Deviation 


i] 


In my school we use 
student results to 
determine yearly goals for 
school improvement. 


17.4% 


In my school student 
results lead to decisions 
with regard to 
professional development. 


13.2% 


I show teachers based on 
data to what extent the 
school is achieving our 
goals. 


Student achievement 
results are used to 
identified gaps in our 
curriculum. 


Detailed data analyses are 
an essential part of 
improvement processes in 
my school. 


63.8% 


In my school we use data 
as a tool to determine 
effective teaching 
methods. 


16.2% 14.7% 


In our school we use 
external evaluations for 
our own improvement. 


The division of teaching 
time in my school is based 
on identified learning 
needs of students. 


57.4% 


Results of students are a 


24.6% 
used to evaluate teachers. 7 


Organizational characteristics that included vision, leadership, and support related to data use were 
measured by 16 items on the School Organization Characteristics Scale. As shown in Table 4, this scale generated a 
trend of positive agreement in responses but with greater variation than the previous two scales. Not a single 
principal disagreed that they were strongly encouraged by the district to use data as a tool to support effective 
practice. Principals responded to the top three statements with high agreement (rating of 4 or 5), indicating a shared 
awareness of the need to develop data analysis skills (96%) and consistent encouragement from leadership to use 
data. Notable for the focus of this study, only slightly more than half the principals (52%) agreed that someone 
helped them change their practice based on data. Less than half agreed or strongly agreed that a specific time was set 
aside for data use (44.9%). 
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Table 4. School Organizational Characteristics Scale (N =70) 

i Strongly Agree Std. 
=5 Mean F deviation 

My school is aware that we 

need to keep developing the 

skills of teachers to analyze 

data. 


Our district encourages data 
use as a tool to support 
effective teaching. 


8 
2 


32.4% 4.28 0.54 


8 
2 


My supervisor encourages 
data use as a tool to support 
effective practice. 


In my school we use a 
structured method to analyze 
and interpret data to base 
actions on. 


Data use is a priority in my 
school. 


There is someone who 
answers my questions about 
using data. 


2 


I am adequately supported in 
the effective use of data. 


8 
3 


Teachers in my school share 
acommon understanding 
about what good teaching is. 


My supervisor creates many 
opportunities (e.g. time) for 
me to use data. 


B 
2 


g 
2 


My supervisor is a good 
example of an effective data 
user. 


2 


" 
2 
& 
2 


Teachers in my school share 
a common understanding of 
what student learning is. 


8 


8 
ee eee eee ne 


My supervisor discusses 
data with me. 


3 
2 


My supervisor discusses the 
results of their data analyses. 


ww 
B 
z 


2 
3 


& 
a 
z 


There is someone who helps 
me change my practice 
based on data. 


Teachers in my school share 
a common understanding 
about effective ways to 
evaluate student learning. 


There is specific time set 
aside for me to use data. 


HBHEHRHRHRHEHHReeHeeeese 
g 


User characteristics of principals that influenced data use including dispositions toward data use and levels 
of perceived knowledge and skill are shown in Table 5. More than 95% of principals agreed (rated 4 or 5) with all 
three items involving the benefits or importance of data use, indicating they believed students benefit when 
instruction is based on data (95.6%), that data is important in determining individual student needs (98.5%), and that 
data is important in changing their own practice (95.5%). 
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Table 5. User Characteristics Scale (N=70) 


Students benefit when 
instruction is based on data. 


It is important to use data in 
determining individual student 
needs. 


Data are important in changing 
my practice. 


I am comfortable in interpreting 
data that are presented in 


graphs. 


I know how to interpret the data 
and the reports I receive (exam 
results, student achievement 
results of previous years) 


I am able to adjust my practice 
based on data. 


I am able to use data to 
diagnose student-learning 
needs. 


I understand the quality criteria 
and concepts for data use (e.g., 
correlation, validity, reliability). 


Following User Characteristics, the next most positive responses were generated by the five items from the 
Computer System Characteristics Scale. Table 6 shows that two of the five items yielded means above 4.0 and the 
remaining three, above 3.5, indicating general agreement that the district’s computer systems met their needs. 

Table 6. Computer System Characteristics Scale (N=70) 


Strongly disagree Disagree Strongly agree Std. 
=1 =2 =5 Mean deviation 


I have the proper technology to efficiently _ 38.2% 4.24 0.77 
examine data. c si ; : 


The computer systems in my district provide me 35.3% 
access to lots of data. ‘ _ 


The computer systems in my district allow me 
to examine various types of data simultaneously | 22.1% 
(e.g., attendance, achievement, demographics). 


The computer systems in my district generate 


displays (e.g. reports, graphs, tables) that are i . a 13.2% 
useful to me. 


The computer systems (for data use) in my 17.6% 
district are easy to use. : . : 


The Data Characteristics Scale consisted of 10 items measuring principals’ perceptions of data quality, such 
as access to timely, relevant, and accurate data. This scale had the lowest overall mean of the four scales on the 
survey (M = 3.33) and the most variation in responses as shown in Table 7. Most principals agreed or strongly 
agreed (rating of 4 or 5) that they had access to relevant student data (85.5%), which tracked progress (81.6%). 
Agreement surrounding descriptors of data accuracy (60%) and timeliness (63.8%) was slightly less with a third 
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responding neutral (rating of 3) or disagreeing (rating of | or 2). When asked whether data was available within 3 
weeks of the beginning of a school year, 20% disagreed and more when students started midyear (27.5%). The 
majority of principals disagreed (rating | or 2) that they could find all the data in one system (67%). 


Table 7. Data Characteristics Scale (N=70) 


Strongly disagree Disagree Strongly agree Std. 
=i =2 =5 Mean deviation 


re 18.8% 3.94 0.82 


70.8% eal 10.8% 


I have access to relevant data on my 
students. 


I have data on the progress of my 
students. 


The data to which I have access to is 
help me plan my support. 

With the data I have on my students, 

I can determine the growth of my 10.0% 
students from year to year. 


65.7% 


The data I have on my students are 
up to date. 


The data I have on my students are 
accurate. 


Data on my current students are 
available at the beginning of each 18.6% fa 15.7% 
school year (within 3 weeks). 


When students start in the middle of 
the school year, their data becomes 4 36.2% 


quickly available. 


58.6% 


I can find all the data on my students 171% ea 14.3% 
in one system. 


18.6% 


I have too little data on my students. ie 21.7% 


Finally, when principals were asked to describe the impact of data analytic tools on their work, a common 
trend emerged around principals saying that they felt data was generally underutilized. Similar to principals around 
the world (Schildkamp & Kuiper, 2010), the principals in this study talked about the use of data to inform decision 
as one of their greatest challenges (Principal 3). As in other studies, the mere availability of data and tools has not 
guarantee use or changes in practice (Wayman & Stringfield, 2006), and the factors impacting data use have 
remained complex and numerous. The researcher observed several ways that the tools were not as useful and usable 
as they could be. Large volumes of data were available, but the tools provided included mostly static and descriptive 
reports that failed to harness the analytic power of contemporary data processing and predictive analytics as seen in 
educational data mining trends. A gap was identified between the theory that educational leaders should be cross- 
analyzing different data types such as demographic, school process, perceptions, and student learning data 
(Bernhardt, 2018) and what the tools actually supported them to do. There was an underrepresentation of tools to 
measure and analyze perceptual and school process data and few tools integrated more than one type of data. 

The qualitative data generated in Phase 2 confirmed many of the findings surrounding the factors 
influencing data use. A summary of the factors influencing data use were summarized and situated in the conceptual 
framework in Figure 3. 
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Tools: Digital Data Analysis Tools 
Computer System Characteristics: 

- Accessibility 

- Integration between systems 

- Functionality 

- User interface 


Subject: Principal Object: Data Outcomes: Insight and Impact 


User Characteristics: Data Characteristics: Analysis ee Bains alin tae 
: ee ata Characteristics: - Comparing - Improved decision-making for 
- Dispositions ; - Quality ~ Planning ‘. school improvement 
- Knowledge and skills - Timely 7 Reporting - Greater alignment of resources 
- Age - Relevant am ne - Impact not as great as it could be 
- Reliable 
- Valid 
- Quantity 
Rules: Systems and Policy Context Community: School, District Division of Labor: Support 
= Federal (NCLB, ESSA, FERPA) School Organizational Characteristics: - Technology Coordinator 
- Funding - Culture - Grade levels 
- District support - District staff 
- Limited time - Peer principals 


- Professional learning 


Figure 3. Factors influencing use situated in the conceptual framework 
Study’s Significance 


The findings from this study can have implications for the development of theory as well as for 
stakeholders such as district leaders and tool designers. This study reaffirms the tremendous importance of user 
experience studies. A need exists for designers of data platforms to work with principals directly with an iterative 
design mindset so that the tools continue to be refined as data sources change, user needs evolved, research expands, 
and theories are updated. For each of the tensions identified in this study, designers can explore tool enhancements 
to support solutions. For example, users stated that they lacked access to training, so perhaps designers can address 
the need by embedding training modules in the tools. Principals often mentioned that they lacked dedicated time to 
engage in data analysis activities, so designers could enhance time-saving features, such as push notifications that 
would send principals data alerts when key thresholds are met instead of waiting for the principal to log in. One 
interviewee expressed his desire to have natural language processing (NLP) capabilities which could have profound 
implications. Novice data users often wish to explore data but are limited by their inability to formulate their 
questions in terms of tool operations (Setlur, Battersby, Tory, Gossweiler, & Chang 2016). An example can be 
found in the Tableau platform featured called Ask Data, which can interpret the intent behind vague questions to 
produce visual results. An application of such tools in education might empower a principal to ask for the “top 
students” or “highest performers” or “maximum score,” and the system could be flexible enough to interpret a wide 
range of terms to query the data. The dream of having a natural conversation with data is being realized in other 
industries and would certainly benefit school leaders who are not trained analysts. In summary, user-centered 
products are enhanced by user research based on user narratives and observed experiences. 

This study also contributed to theory by creating a pictorial representation of principals’ data analysis 
conceptualized in activity theory to show the interplay of related factors while highlighting the important mediating 
role of the tools themselves (Figure 1). Other frameworks, such as Schildkamp’s data use framework (2012), have 
considered the enablers and barriers of data use to include organization, data, and users but have not emphasized 
technology as a significant factor. The findings from this study confirmed technology as mediator influencing 
educational data use. This study adds to the growing body of research that confirms the usefulness of activity theory 
to guide the development of educational research questions, conceptualize relationships between factors, and 
identify tensions in a school setting. By simultaneously drawing from the field of UX, the researcher was able to use 
the activity theory framework in this study to organize factors that impacted users in a way that can be useful to 
others interested in linking these fields of study. 
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Abstract 


This paper presents the results of a two-year educational design research (EDR) initiative focused on the 
challenge of increasing learners’ motivation and engagement in an instructional design (ID) course. Class 
observations and student evaluations from prior semesters clearly showed the need to redesign a foundational 
graduate course at a large university in the southwestern US. Following the principles of EDR, a team of three 
faculty and two graduate assistants addressed this need through multiple cycles of analysis, development, testing, 
and refinement. The literature review conducted during the analysis and exploration phase of EDR indicated that 
gamification as a pedagogical technique could improve student interest and engagement. Gamification elements 
were introduced to the course design during the design and construction phase and were subsequently tested and 
refined during the evaluation and reflection phase. Data collected included notes and drawings created during the 
research team’s brainstorming sessions and from the course archives on Blackboard. The archives included text 
from various course discussion forums, assignment content, messages from students, and feedback from students 
about their perceptions of the game elements. The redesigned course has been offered for two iterations, and 63 
graduate students have completed the course. Five design principles that are relevant in real-world educational 
contexts emerged through the systematic introduction of various interventions and adjustment of the components of 
the educational content of the course. This EDR study demonstrates how technology-based gamification elements 
could be used to enhance student engagement and motivation and can serve as a model to inspire others who may 
consider gamifying their online and blended courses. 
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Introduction 


Students’ interest and active engagement in a course are critical for learner academic achievement (Soffer 
& Cohen, 2019). Through class observations and student evaluations from prior semesters, the course instructor of 
an instructional design (ID) course at a large university in the southwestern US found that some of her students 
perceived the course to be dry and uninteresting, struggling to stay engaged with the content in the first half of the 
course. Following the principles of educational design research (EDR), a team of three faculty and two graduate 
assistants worked through multiple cycles of analysis, development, testing, and refinement to redesign the course 
with the aim to improve students’ interest, motivation, and early engagement in the content. This case study reports 
findings from this two-year educational EDR initiative and describes how game elements of fantasy storyline, 
avatars, gamified points structure, badges, leaderboard, and interactive multimedia supported immersive play 
experiences within the online course. 


Background 


Smith and Ragan (2004) define instructional design (ID) as a systematic and reflective process of 
translating learning principles into instructional content. Having a good understanding of the instructional design 
process can help educators create courses that are more likely to be effective and engaging. Research studies have 
shown the positive impacts of effective instructional design in many professional education settings, such as in 
medical training (Melo, Falbo, Muijtjens, Vleuten, & Merriénboer, 2016), librarian education (Mullins, 2014), and 
preservice teacher training (Kumar & Hamer, 2012). 

Given the importance of instructional design for students pursuing a degree in instructional technology, a 
foundation ID course has been one of the core courses of our Learning, Design, and Technology graduate program 
since the program’s inception in 1979. The course, entitled CUIN 7390: Instructional Design, introduces students to 
the theoretical, experiential, and critical components of the instructional design process. In addition to core 
knowledge, students gain practical experience by designing stand-alone instructional modules that can be used in 
real educational settings (“Course Descriptions,” 2017). The design of the course adopts the systematic design of 
instruction model introduced by Dick, Carey, and Carey (2014) and uses the textbook written by these authors, The 
Systematic Design of Instruction. 

The initial design of CUIN 7390 followed each phase of the Dick and Carey instructional design process 
and was structured such that each week’s instruction corresponded to a step in the design process. The course began 
with the analysis phase of instructional design (e.g., needs assessment, learner analysis, and performance objectives) 
that would eventually lead to students developing instructional materials for their individual projects. At the end of 
the course, the students formatively evaluated the materials that they created with a sample of their target population 
(Figure 1). 


Conduct 
Instructional 
Analysis 


Revise 
Instruction 


Assess Develop and Design and 


Needs to Wate Develop Beer Select Conduct 

: Performance Assessment Instructional . ; 
Identify Objectives inetrimenté Strats Instructional Formative 
Goal(s) 4 ey Materials Evaluation 


Analyze 
Learners and Design and 


Contexts Conduct 
Summative 


Evaluation 


Figure 1. Dick and Carey's systems approach model. Adapted from The Systematic Design of Instruction (8th ed., p. 
1) by W. Dick, L. Carey, L., J. O. Carey, J. O., 2018, New York, NY: Pearson. Copyright 2018 by Pearson. 


Since this course relies heavily on information in the textbook, students often reported that the large 
amount of reading was challenging. Student evaluations for the past ten years indicated that while the students found 
the overall quality of the course high, they sometimes remarked comments about the content such as, 
“Overwhelming at first!” (Anonymous student response on an end-of-course evaluation). Some students over the 
years described discomfort and stress that they felt in the first part of the course before they understood the ID 
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process as a whole. For example, one student commented, “In the beginning, there was no light at the end of the 
tunnel. It has, however, come full circle and made more sense” (anonymous student response on an end-of-course 
evaluation). In preliminary redesign discussions, the primary instructor shared that students who were organized, 
methodical, and self-regulated tended to engage with the course materials early in the semester and overall 
performed well in the course. However, she said that other students who were less detail-oriented, auditory, or visual 
learners appeared to find the systematic process dry and uninteresting and would often struggle in the first half of the 
course. Such students would typically have greater interest and engagement once they realized the big picture and 
how the planning phase of the ID process fit into the development phase as they started to put the pieces together 
about halfway through the course (Personal communication with the primary course instructor). Since this course is 
usually offered entirely online, we saw a strong need to redesign the course and apply innovative instructional 
strategies that could motivate students and engage them with the content earlier in the semester. We also felt that 
this course was a good candidate for applying gamification elements for three main reasons — 
1) This course is an introductory course, but it is very important because students learn essential skills 
they will need in other courses in the program; 
2) Because this is a theoretical course, the instructional design content did not change significantly from 
year to year; and 
3) The course is taught every spring, enabling the research team to quickly evaluate what worked and 
what was needed to be revised for the next iteration the following year. 


Educational Design Research 

To address the need to enhance student motivation and engagement, we followed the principles of EDR to 
generate insights through multiple cycles of analysis, development, testing, and refinement (McKenney & Reeves, 
2019). Central to EDR is the iterative testing and refinements on complex educational problems to derive evidence- 
based claims that may potentially impact naturalistic settings (McKenney & Reeves, 2019). When doing EDR 
(sometimes referred to as design-based research or DBR), researchers systematically introduce various interventions 
or adjust the components of their educational content to refine and produce practical knowledge or theories that are 
relevant in real-world educational contexts (Barab & Squire, 2004; Brown, 1992). DBR studies evaluate “what 
works; for whom; under what authentic, field-based conditions; and how/why this approach is effective” (Dede, 
Ketelhut, Whitehouse, Breit, & McCloskey, 2009, p.14). 

Although it is not a linear step-by-step process, EDR typically involves three major phases. The initial 
analysis and exploration phase involves working closely with collaborators to acquire an understanding of a 
significant educational problem and investigate how others have addressed it. This phase gradually morphs into a 
design and construction phase that is focused on identifying or creating appropriate design principles and using 
these principles to develop a prototype intervention that addresses the problem. Finally, an evaluation and reflection 
phase of multiple iterations of data collection and analysis is conducted to test the prototype intervention and review 
the implications of the findings. Figure 2 depicts the EDR process. 


Implementation & Spread 


- a“ a” = 
¥ ¥ ¥ ¥ 
Maturing 
Analysis Design Evaluation ‘ 4 Tile-ta-lalelelal 
al > al > 
Exploration Construction Reflection —_ Theoretical 
Understanding 
“A A a 
—w 


Figure 2. The process of conducting educational design research. From Conducting Educational Design Research 
(2nd ed., p. 83) by S. E. McKenney and T. C. Reeves, 2019, New York, NY: Routledge. Used with permission. 
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Gamification 

The literature review conducted during the initial analysis and exploration phase suggested that 
gamification as a pedagogical technique could improve student interest and early engagement (Betts, Bal, & Betts, 
2013; Gibson, Ostashewski, Flintoff, Grant, & Knight, 2015). The term gamification refers to the “use of game 
design elements within non-game contexts” (Deterding, Dixon, Khaled, & Nacke, 2011, p. 1) and involves the use 
of “game-based mechanics, aesthetics and game thinking to engage people, motivate action, promote learning, and 
solve problems” (Kapp et al., 2014, p. 54). While the application of gamification in education is still an emerging 
trend, its proponents suggest that it can be employed to enhance student engagement and prompt learning (Dicheva 
et al., 2015). 

There are some elements that are commonly found in a gamified course: Story, rules, challenge, curiosity, 
character, interactivity, feedback, and freedom to fail (Buckley, Doyle, & Doyle, 2017; Kapp et al., 2014). It is not 
necessary to add all of these elements when doing gamification on a course. However, Nicholson (2014) suggested 
the use of “narrative, freedom to choose paths to explore, playful activities, and opportunities to reflect” (p.14) for a 
gamified course to be engaging and meaningful for students. 


Methodology 

McKenney and Reeves (2019) suggest that EDR is not so much a specific research methodology as it is a 
genre of educational inquiry that seeks to maximize the practical impact of educational research while at the same 
time seeking to reveal new theoretical knowledge. For our study, we pursued EDR for two main reasons: (1) the 
parallel resemblances of its process with the ID process and (2) the naturalistic or real-life context emphasis of the 
method. Central to EDR is the iterative testing and refinements on complex educational problems to derive 
evidence-based claims that may potentially impact naturalistic settings (McKenney & Reeves, 2019). When using 
EDR, researchers systematically introduce various interventions to or adjust the components of their educational 
content to refine and produce practical knowledge or theories that are relevant in real-world educational contexts 
(Barab & Squire, 2004; Brown, 1992). EDR generates insights through its multiple cycles of analysis, development, 
testing, and refinement, from which new insights would be generated to improve the following cycles. 


Research Questions 
Our EDR initiative addressed two overarching research questions — 
1. What is an optimal design for applying gamification elements in an online instructional design course to 
improve student motivation and increase their engagement as early as possible in the course? 
2. What design principles can be identified that can be used to extend the application of gamification elements 
in other online or blended courses? 


Data Sources 

The data for this study included notes, drawings created during the research team’s brainstorming sessions, 
and reflection notes taken during and after the end of each course. We also examined the course archives on the 
Blackboard learning management system (LMS). The archives included text from various course discussion forums, 
assignment content, messages from students, and post-course feedback from students about their perceptions of the 
game elements. 

This paper reports findings from two iterations of the redesigned ID course. In Year One, 24 students 
completed the course, and 39 students completed in Year Two. The students’ ages ranged from 23 to 54 (M=6.67, 
SD=8.94) with 20.6% identified as male and 79.4% female. At the beginning of the course, we asked students to 
complete an online survey that was modified from the student course engagement questionnaire (SCEQ) 
(Handelsmann, Briggs, Sullivan, & Towler, 2005). The survey was optional and not tied to their grades. The SCEQ 
is well-validated to measure student engagement and has been reported to have high-reliability scores (Ginns, 
Prosser, & Barrie, 2007; Handelsmann et al., 2005). The Cronbach alpha for this study was .832. The 23-item survey 
uses a 5-point scale ranging from | (not at all characteristic of me) to 5 (very characteristic of me). This survey 
measure student’s perception of course engagement in four categories — skills, emotions, participation, and 
performance. Several of the items were slightly reworded to match the course settings (see Appendix 1). For 
example, item 3 — “Doing all of the homework problems” substituted “homework problems” with “assignments.” 
Table | displays the SCEQ results for students who took the course in Years One and Two. 
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Table 1. Student course engagement questionnaire scores for the first and second years 


Year Skills Emotional Participation Performance 
first year Mean 38.31 20.63 24.50 14.5 

N 16 16 16 16 

SD 3.790 2.604 3.688 1 
second year Mean 37.80 20.97 22.77 13.55 

N 30 31 30 31 

SD 3.478 3.114 3.748 1.36 


Educational Design Phases 


Phase 1: Analysis and Exploratory Phase 

In the fall of 2016, our team began the redesign process by reading and discussing the book, The 
Gamification of Learning and Instruction Fieldbook: Ideas into Practice by Kapp et al. (2014), as well as research 
articles about educational gamification. We held weekly meetings to discuss our ideas and brainstormed ways of 
making the game more engaging and authentic. We discussed the design of games that we had experienced, 
including older immersive games, such as Myst (Miller & Miller, 1993). 

Choosing a story for the game was a critical first step in the process. Kapp et al. (2014) suggest that stories 
could help improve learner’s engagement, since they “evoke emotions, provide a context for placing information, 
and are the way humans have handed down information for centuries“ (p. 236). Because of our proximity to Johnson 
Space Flight Center in Houston, Texas, we decided to use a space theme for the storyline. Throughout our 
brainstorming sessions, we discussed the script, characters, challenges, and possibilities for adding more 
interactivity to the course content. We also created sketches and flowcharts for ideas that could carry the content 
through the 15 weeks of the semester and still convey the essential ideas and skillset of the ID process (Figure 3). 
This spurred exploration into space terminology and content as we developed the prototype during the next EDR 
phase. 
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F igure 3. Drawings show the evolution of the Instructional Design Game Board from concept to reality. 


As ID process can be easily divided into five components, the research team decided that the space “crew” 
(the students) could visit five planets on their quest to find an “evil villain” who stole important ID documents. 
Planet names were based on Latin words that describe the focus of ID process component. For example, the first 
planet was named Resolvere (meaning “to analyze’’), since identifying instructional goals using front-end analysis 
and analyzing learners and context were covered in this section. Students would have missions to carry out on each 
planet that would culminate with their completing their own ID project, and they worked individually and in teams 
through the ID process to create design documents for their content area. 

We explored the idea of using commercial software to create the game, and we researched companies who 
developed these tools. We contacted two companies who had received recent design awards for their game software 
and evaluated demos of their programs. Commercial software appealed to us at first, because it offered many 
features including built-in leaderboards, avatars, and badges. However, the more behavioral aspects of the game 
software, as well as the high cost, deterred us from using this software and resulted in our developing the game 
using a variety of tools including Adobe Captivate and Photoshop, Camtasia, and features already existing in Google 
Sheets and Blackboard. 


Phase 2: Design and Construction Phase 

We began the design and construction phase by chunking the course content and aligning it with the 
storyline using a Google Doc shared with the team. This allowed team members to interact continuously by leaving 
comments and editing the text (Figure 4). 
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Intro: 
The year is 2288 


5 months ago, a rogue spaceship captain has stolen all of the instructional design materials on Earth. 


Susie Gronseth 
Schools around the world are falling into chaos without the ability to properly design instruction. Jan 11, 2017 


If we do not recover the instructional design materials soon, the earth’s future will be lost. They wont all be leading, right? They 
will be a part of an international crew.... 


You have been called in by Commander McNeil George Zhao 


17 
You have been chosen to lead an international crew into space a 


They each have their own ship. The 

international crew part refers to 

And to pursue and apprehend the dastardly Captain Beltran. navigator Zhao, science officer 
Handoko, and engineer Zhang who 
accompanies them on their mission 


On an expedition to recover the stolen instructional materials 


Figure 4. Google doc used to align, edit and revise the storyline, script, and story videos. 


Music, ideas for graphics, and scripts for the audio were also added to this document as the design was 
finalized. As the construction of the course began, we also added a Google Doc to keep track of changes needed and 
completed (Figure 5). 

4+—-Update power-up ehattenge Complete 
a. Need to take out GSFC, and maybe add some examples from last year? 
2—Cheelcaterta -quiz te make sure itis readyte-ge-Complete © Susie Gronseth 


a. Ran an item analysis on Quiz 8/9 - Question 8/9-06 was missed most. Suggest Mar 6, 2018 
we add a table that reviews strategies for different types of skills to Materia 
content (see example here: 


http://faculty.mercer.edu/codone_s/tco665/2014/weeknine.html) 


Added example from Ngozi. Requested 
her permission via email. 


. Ran an item analysis on Quiz 10 - Question 10-07 was missed most (9 out of 23) 
but seems clear to me. Question 10-04 about the role of the ID was missed next Susie Gronseth 
(8 out of 23), but will need to think about this one. Mar 6, 2018 
: de a ORE WIT ON Re a Added this link to the Readings, Videos, 
. Reetamater-gametertaterte Complete 
. Update-eeurseteadingsteseurees Complete 
. Viermbhete-gameterSpringBreak Complete 


and Resources for Planet Materia 
section. 


Figure 5. Tracking changes needed and completed in the course using a shared Google doc. 


As we worked on the storyline and learned more about space exploration through conversations with space 
subject matter experts, we added elements such as a “wormhole” for “disappearing” during spring break, used a 
photo of the mission control room at NASA as the background for the Commander (instructor) videos, and created 
assignments for small groups to create educational materials for the actual eclipse that would take place the summer 
after the revised course was offered. On each planet, we planned interactive multimedia elements for the students 
that would reveal the next chapter in the “Captain Beltran” story and enable them to “unlock” the next section of 
course materials by locating passcodes and completing content-related tasks within the game. Course instructors 
became facilitators, and they would communicate through instructional videos, supportive handouts, and feedback to 
guide the students throughout the course. 

When we began discussing how we would construct the game interface, we realized that we could not carry 
out the complete vision of our online course experience solely using the functions of Blackboard. This challenged 
our team to identify and bring in new tools, such as Voki for creating the student avatars and talking avatar 
“explainer” videos, Google Sheets for running a live class leaderboard with a linked “behind the scenes” instructor 
sheet, and Camtasia and Adobe Captivate for generating interactive multimedia game-play. We transformed our 
online discussions from a post once, reply twice expectation to a team-driven, goal-oriented discussion framework. 
We found that students were so invested in their teamwork that they posted more actively and initiated even further 
discussion through student-created Google Docs, Hangouts, and Slack channels. 

Much of our design discussion centered on how to transform the previous points structure of the course into 
the game format. Students accumulated “flight hours” (points) for completing missions, quizzes, and bonus “Power 
Up” challenges and for actively participating in discussions. The “space crew” leaderboard showed students’ current 
standings with regard to flight hours earned, game levels achieved, and badges earned. Students were represented 
by avatars and the game names they created, which were not their real names, enabling them to assess how they 
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were performing against the other team members while maintaining their anonymity. The leaderboard was easily 
accessible to students through a link in the main menu of the Blackboard course, and we recognized the current 
leaders on each planet in course announcements. We also designed three different badges that were displayed on the 
leaderboard. One badge, Patches Earned, represented the student’s current astronaut rank earned. Students began 
the course as Junior Astronauts and then rose to higher positions as they earned “flight hours” (points). Another 
badge that was incorporated into the leaderboard were wings that were also aligned to students’ current rank. The 
third badge displayed any points earned from the “Power Up” challenges, which were bonus activities for each 
planet. Figure 6 shows a portion of the leaderboard with student-created avatars and names, as well as the three 
types of badges, “total hours in space” (accumulated points), level, and hours needed to move up to the next level. 


You are ‘ b. 
aT THE : Master 
HIGHEST 4 ASTrOnNauT 
Lever 


ASTria ROSeCBUD LeveL. S 


You are 
aT THE Www MasTer 
HIGHEST ASTTOnNauUuT 
Lever 


LeveL S 


senior 


ves aAsTronaut 


Figure 6. Course leaderboard. 


Another complex task involved tracking the different points that students could earn while on each planet. 
Figure 7 shows a screenshot of the activities for one planet in the tracking system we used to assign points and 
bonus points for the challenges. 


Points/Flight 
Hours 


Planet Resolvere 


Week 2 Engagement 
e Planet Resolvere Game (game SCORM) 
(1 pt.) 
e Planet Resolvere Mission video (1 pt.) 
e Week 2 Discussion (3 pts.) 


Planet Resolvere Mission 
Goal Statement(s) (4) 
e Description of Learner Interview (2) 
e Learner Analysis (3) 
e Context Analysis (6) 


| Bonus: Power Up Challenge? Power Up | Bonus: Power Up Challenge? 1 


Figure 7. Screenshot of the online tracking system for possible points on Planet Resolvere. 
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Lastly, throughout the process, we also created short videos to provide an overview of each planet and 
introduce the “mission” (assignment) that they had to complete on the planets. We also created interactive activities 
on each planet. 

The gamified course was first implemented in spring semester 2017. Figure 8 shows the course schedule 
for the first implementation of the course in spring 2017. 


es 


Getting Ready: Space Camp! 
1/16 - 1/21 Ch. 1: Introduction to Instructional Design 


1/22 - 1/23 “Flying to Planet Resolvere” / Recharge 


Planet Resolvere (“Analyze”) 


1/24 - 2/4 Ch. 2: Identifying Instructional Goals 
Ch. 5: Analyzing Learners and Contexts 
2/5 - 2/6 “Flying to Planet Artem” / Recharge 


Planet Artem (“Skills”) 


2/19 - 2/20 “Flying to Planet Consilium” / Recharge 


BY 2/7 - 2/18 Ch. 3: Conducting a Goal Analysis 
Le Ch. 4: Identifying Subordinate and Entry Skills 
ch 


Planet Consilium (“Design”) 


2/21 - 3/4 Ch. 6: Writing Performance Objectives 
Ch. 7: Developing Assessment Instruments 


3/5 - 3/6 “Flying to Planet Materia” / Recharge 


Figure 8. Screenshot of course schedule for 2017 spring semester showing the activities on each planet. 


Phase 3: Evaluation and Reflection 

After the first implementation of the course, the team met to reflect on the experience. We shared student 
comments we had gathered throughout the course. Overall, debriefing data centered on the learning of new 
technology skills. There was a feeling of accomplishment about how we had created a vision for the course and 
designed innovative ways to make it a reality. For example, since we did not use commercial software to create the 
game, we had to design our own leaderboard from scratch with Google Sheets and embedded graphics using 
formulas and displays that we learned how to use during the construction phase. We also used Blackboard’s 
advanced features such as SCORM and adaptive release of content. In addition to technology skills, we also 
acquired new pedagogical skills, as we spent time in the design and construction phase trying to create activities that 
would challenge students, while, at the same time, allowing them to practice ID. 

The team noted two areas that needed improvement after the first-year implementation. In Year 1, the 
students were divided into small groups; in these small groups, students followed a simplified ID process and 
created educational materials about the eclipse which would take place in August 2017. We felt this would give 
students a chance to support each other and learn how to use the ID process collaboratively. However, there were 
too many things happening at the same time; students were working on their own ID projects as well as their group 
project while they were also on a journey through space and learning about the ID process. In Year 2, we changed 
the assignments in the small groups so that students shared their progress on their own ID project with their group in 
order to give and receive peer feedback. 

At the end of the course, we asked our students to provide voluntary feedback regarding the structure of the 
course and its game elements. We designed this survey to gather input about components of the course that they 
perceived helpful in motivating and keeping them engaged with the course content and components that they felt 
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could be improved. The survey is composed of sixteen items, with fourteen Likert-type items, one rank-order item, 
and one open-ended item. Table 2 shows student post-course survey responses for the Likert-type items. 


Table 2. Descriptive Statistics Results from the First and Second Iterations’ Post-Course Survey. 


first iteration second iteration 

Item N Mean SD N Mean SD 

1 The gamification elements in CUIN 7390 made it 16 = 3.88 1.147 31 3.74 1.182 
feel like a real game. 

2 In general, how effective was the gamification 16 93.31 1.250 31 3.19 1.108 
approach used in CUIN 7390? 

3 Your motivation in this gamified course 16 93.75 931 31 3.90 .790 

4 Your engagement in this gamified course 16 = 4.00 1.033 30 3.80 .805 

5 The amount of work overall in this gamified course 16 = 3.88 719 30 3.77 .728 

6 The difficulty of this gamified course 16 (3.31 .704 31 3.39 715 

7 The ease of navigating the course materials in this 16 93.31 1.138 31 3.42 .720 
gamified course 

8 Time spent reading and viewing course materials in 16 3.63 957 31 3.71 .643 
this gamified course 

9 Time spent discussing course content with 16 = 3.50 1.155 31 3.39 .667 
classmates in this gamified course 

10 Time spent completing the assignments in this 16 = =3.69 .704 31 3.84 583 
gamified course 

11 Overall, how useful were the course activities in 16 4.00 .966 30 4.07 .980 


helping you become more knowledgeable in 
instructional design? 


12 The textbook and supplemental learning materials 16 = 4.13 1.258 31 4.06 854 
used in the course were: 


13 The instructions for the weekly course activities 16 4.44 512 31 4.52 .508 
were: 
14 This was a worthwhile learning experience. 16 4.37 1.025 31 4.77 497 


Design Principles for Gamifying an Online ID Course 
The evaluation and reflection phase of this EDR project revealed design principles related to the integration 
of gamification elements into an online ID course. 


Principle 1: Apply a Team Approach in Course Gamification 

The redesign of CUIN 7390: Instructional Design was a collaborative effort of a team of three faculty and 
two graduate students in the Learning, Design, and Technology area. We began working on the project at the 
beginning of Fall 2016 and went through the iterative process of EDR to first clarify the problem, then create and 
test practical interventions, and finally engage in reflection to reveal theoretical insights to improve student interest 
and engagement. Once the literature review indicated that gamification might be a viable solution for increasing 
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student motivation and engagement, the team members read about gamification in The Gamification of Learning 
and Instruction Fieldbook: Ideas into Practice (Kapp, Blair, & Mesch, 2014). We also studied published research on 
how gamification has been applied in online courses, and reviewed digital tools that could support various game 
elements. Analysis of our design notes made it evident that the team reexamined and redefined in each iteration the 
approach in which the course was delivered to better connect with the learners and explore the content more deeply. 


Principle 2: Use Story to Reinvigorate and Provide Continuity in Online Course Content 

Viewing the course through a new lens for teaching online using gamification provided inspiration for the 
designers to “play” with teaching ideas, providing a refreshed perspective on content that had been previously taught 
in much the same way for years prior. Story and role-play elements were incorporated in the redesigned course 
through the space exploration game segments and learner avatars and screen names. The “space crew” leaderboard 
showed students’ current standings in regard to flight hours earned, game levels achieved, and badges earned. 
Students completed tasks individually, as well as worked in teams to design informal instruction for a simulated 
NASA project. Story became a powerful way to engage students in learning a potentially intimidating process of 
systematic ID. 


Principle 3: Integrate Multimedia Tools to Support Gamification Application 

The complete vision of the gamified online course experience could not be accomplished solely using the 
functions of the LMS. The team used Google Docs for online collaboration in the design process, identified image 
and video creation tools for creating avatars and talking avatar explainer videos, created cloud-based spreadsheets 
for the live class leaderboard, and used Adobe Captivate for generating interactive multimedia game-play. Online 
discussions were transformed to be team-driven and goal-oriented. It was found that students were so invested in 
their teamwork that they overall posted actively and initiated even further discussion through student-created 
documents, synchronous sessions, and asynchronous communications. 


Principle 4: Integrate Interactivity in Gamification 

A key outcome of this project has been maximized student academic learning time. We found that 
embedding interactive online multimedia throughout the course modules immersed learners in a powerful story and 
provided opportunities for them to acquire, practice, and apply ID concepts and skills. They increased their time 
exploring materials in the course site, often completing optional activities above and beyond the required 
assignments. The novel approach to grading through the accumulation of “flight hours” for successful completion of 
activities made it possible for students to extend their learning in directions that were meaningful and useful for 
them and served as a motivator to encourage them to reach increasingly higher levels in the course. 


Principle 5: Provide Opportunities for Peer-to-Peer Interaction 

Even though students were separated by physical distance, the team-based discussions relating to the 
development of the solar eclipse instructional units connected students to each other and the content through shared 
design work and contextualized their learning within a realistic ID case. The course content that may have felt at 
times complex and difficult to understand became more palatable, as our redesigned course offered an abundance of 
varied examples and practice with visual, auditory, and hands-on elements aimed at helping students grasp the 
essence of the ID process. The examples and practice experiences were tied together through the space adventure 
storyline, demonstrating to students that ID can be playful and creative. 


Discussion 


This study has demonstrated how technology can be used to enhance student engagement in online 
environments and to serve as a model to inspire others who consider applying gamification to their online and 
blended instruction. Some might ask why we used EDR to address the need to redesign the ID course instead of the 
systematic design of instruction model (Dick et al., 2014) taught in the course. ID models are adequate if the 
primary goal is to develop a more effective course alone, but we sought to accomplish the twofold mission of EDR, 
which is to solve a real-world problem (enhancing student motivation in online learning) and to identify new or 
refined theoretical knowledge (reusable design principles related to the use of gamification elements in online or 
blended courses). 

The course discussed in the study incorporated game elements of a fantasy storyline, role-play, gamified 
points structure, badges, leaderboard, and interactive multimedia that support immersive play experience. It shows 
the variety of ways that content can be represented, organized, and presented to the learner through graphics, audio, 
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video, text-based handouts, and hands-on explorations. Course designers may apply the design principles that 
emerged in this study as an inspiration for gamification of online and blended courses in other instructional contexts. 
Future research may focus on specific game elements that can be used to support an immersive play experience. 
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Appendix 1. 
Online Student Engagement (OSE) Scale (Handelsmann et al., 2005) 


Within that course, how well do the following behaviors, thoughts, and feelings describe you? 
Please answer using the following scale: 

. not at all characteristic of me 

2. not really characteristic of me 

3. moderately characteristic of me 

4. characteristic of me 

5. very characteristic of me 


— 


Skills subscale: 

1. Making sure to study on a regular basis. 

2. Looking over class notes to make sure I understand the material. 
3. Being organized. 

4. Taking good notes. 

5. Listening carefully. 

6. Coming to every class (face-to-face or online). 


Emotional subscale: 

7. Putting forth effort. 

8. Finding ways to make the course material relevant to my life. 

9. Applying course material to my life. 

10. Finding ways to make the course interesting to me. 

11. Thinking about the course between class meetings or online class activities. 
12. Really desiring to learn the material. 


Performance subscale: 

13. Doing all of the assignments. 

14. Getting a good grade. 

15. Doing well on different class assignments. 

16. Being confident that I can learn and do well in the class. 


Participation subscale: 

17. Raising my hand in class or participating in online class activities. 

18. Asking questions when I don’t understand the instructor 

19. Having fun in class. 

20. Participating actively in small-group discussions. 

21. Using discussion forums or talking to the professor outside of class to review assignments or ask questions 
22. Helping fellow students. 
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Abstract 


This quantitative study investigated the use of Visual Thinking Strategies (VTS) in a higher education course for 
critical thinking. Participants (n=18) were exposed to VTS through weekly exercises. An instrument was developed 
to evaluate the participants’ responses based on Abigail Housen’s Stages of Aesthetic Thoughts (Housen 1999). 
Their written pre- and post-treatment responses were compared to measure growth in their critical thinking skills. 
Key findings support growth in both the quantity of basic and the inclusion of higher level observations. Previous 
research has found that training in VTS increases critical thinking skills which transfer to the individual’s domain, or 
other areas of study (Housen 1999). 


Introduction 


Development of critical thinking skills is one of the tenets of higher education (Yenawine & Miller 2014). 
Visual thinking has developed as a field within psychology using aesthetic viewing as a strategy to increase critical 
thinking skills. This study investigates a higher education course where Visual Thinking Strategies (VTS) is used to 
develop critical thinking skills by viewing and analyzing artwork. As Arnheim puts it, “Identifying what we see is 
an act of cognition” (Visual Thinking Strategies 1997 p. 2). 

Visual Thinking Strategies is an observational process with a facilitator asking three leading questions 
about a visual example. The facilitator summarizes responses without evaluation, and builds the conversation from 
thestatements. VTS has shown to be effective in evoking participation among learners (Housen 2001). It has been 
utilized in museums as well as in elementary classrooms to improve participants’ critical thinking, visual 
understanding, and communication skills (Reilly, Ring, & Duke 2005). 

Although VTS has been tested on elementary students (Housen 2001) and medical and nursing students 
(Reilly et al. 2005; Klugman, Peel, & Beckmann-Mendez 2011), there has been little research on the effectiveness of 
VTS on general college undergraduate students This study assessed the growth of visual critical thinking skills of 
undergraduate students (n=18) from the first week to the ninth week of the term. Students were in a semester-long 
course on visual and critical thinking at a major Midwestern university. 

An instrument was developed for this study to measure the participant’s critical thinking based on Housen's 
Five Stages of Aesthetic Viewing (Housen 1999). Participants completed a VTS Exercise, writing their thoughts 
regarding Hopper paintings on the first and the ninth weeks of class, which were evaluated with the VTS instrument. 
Weekly verbal exercises were presented in class between these evaluations to familiarize the participants with VTS. 
This study measures whether there was an observable increase of critical thinking skills between week | and week 9. 
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Motivation and Significance 

VTS can be used in higher education to increase effective learning and student engagement (Yenawine & 
Miller 2014). VTS’s structured, open-ended discussions emphasize that there may be multiple correct perspectives 
to support an argument. VTS is also a tactic used to approach unfamiliar topics with peer collaboration. 

As Philip Yenawine and Alexa Miller put it, information in the 21 century is “complex, ambiguous, 
changing, and requiring synthesis across disciplines” (Yenawine & Miller 2014, p. 5). For students, it is critical for 
them to be able to deal with ambiguity, uncertainty, collaboration, and rapid changes. VTS better prepares students 
through the nature of its progression: observation, asking questions, presenting an argument with evidence, 
teamwork, and critical thinking. In addition, VTS has been observed to be helpful for English-second-language 
learners and students with learning challenges as it encourages oral communication and has a facilitating figure 
rephrase what the student had said, potentially clarifying what the student meant to say. (Yenawine & Miller 2014). 

The use of art as the visual artifact for observational study in VTS is valuable to further critical thinking 
skills. "The nature of artwork is ... ambiguous in meaning, multilayered, intentionally open to interpretation, and 
often have symbolic and abstract elements; making sense of them offers great training for our minds” (Yenawine & 
Miller 2014). Visual Thinking Strategies can also be used to evaluate non-art based visual material. 


Literature Review 

The following section will go through the history of VTS, the Developmental Theory, the five Stages of 
Aesthetic Viewing (Housen 1999), prior K-12 student research on VTS, and prior Medical and Nursing student VTS 
Research. Through this section, the importance of this VTS research as well as the gaps in VTS that have not been 
studied are explained. 


History of VTS 

Beginning in the 1970’s, Abigail Housen searched for methods to increase viewer engagement with 
aesthetic work, and her research demonstrated that students learn best through active learning (Housen, 2001). She 
collaborated with Philip Yenawine, who was a museum educator at New York’s Museum of Modern Art, to find a 
way to effectively teach and measure viewing skills (Yenawine 2013). 

Housen and Yenawine designed a method to view artwork in a group environment led by a facilitator. The 
facilitator asks simple, but thought-provoking questions to promote discussion, which are: a) "What is going on 
here?" b) "What do you see that makes you say that?" and c) "What more can you find?" (Housen 2001). The 
facilitator asks these questions, summarizes responses, points at the specific parts of the artwork that is mentioned 
and does not judge or evaluate the response (Housen 2001). 


Developmental Theory 

Learning is most effective when the learner actively participates and is given a chance to reflect on the 
material. Regardless of age, individuals feel motivated to overcome the challenges in their lives. Thus, the 
environment that one inhabits can influence how much, how well, and how quickly he or she learns. (DeSantis & 
Housen 2011). These findings are important bases for Housen's Aesthetic Stages. She used Vygotsky's and Piaget’s 
theories as the motivation for her study. She uses verbal comments to understand individuals' stage of aesthetic 
thought. The Aesthetic Stages are progressive. Individuals of lower Stages of Aesthetic Viewing cannot understand 
the art at the same depth as upper stage viewers (DeSantis & Housen 2011). 


Five Stages of Aesthetic Viewing 

Abigail Housen developed the five stages of aesthetic viewing after interviewing and categorizing various 
individuals. She had noticed a pattern emerging from the interviewees’ thought processes as they examined the 
artwork. The stages are summarized below. (DeSantis & Housen 2011; Housen 1999) 


[1] Stage I: Accountive 

Viewers in this stage tell a story of what they believe is occurring based on their personal experience and 
emotions. They connect the solid pieces that they see within the image and create a narrative out of this that makes 
sense to them. 


[2] Stage II: Constructive 


Viewers bring their past experiences and knowledge to understand the image. Viewers may dismiss an 
image if what is depicted is not an accurate representation of the natural world or if high-quality craft is not shown. 
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Viewers also begin to show signs of interests for the artist’s intentions and the message that is attempted to be 
communicated. 


[3] Stage HI: Classifying 

At this stage, the viewers begin to view the art from the perspective of an expert, being critical and 
attempting to understand the artwork's message by using the techniques and skills that they have learned. 

[4] Stage IV: Interpretive 

Viewers in this stage let the artwork express itself and tell its story. Viewers appreciate the small details of 
the work and use their critical skills to help understand the work. They are aware that their perception and the 
feelings that they get out of the work will differ depending on the situation. 


[5] Stage V: Re-creative 

By this stage, the viewer has spent a long time observing and analyzing works of art and may be a 
professional art historian or a professor in a related field. They are an expert in the work, knowing the details such as 
"its time, its history, its questions, its travels, its intricacies" (Housen 1999). 


This clarification of the different Aesthetic Stages is important as this research also involves categorizing 
the participants (students) thoughts of this study into these stages. The participants are not expected to be above the 
third aesthetic stage as that the expertise of a professional. For example comments by individuals of each stage, see 
appendix A. 


VTS Research 

Housen conducted a five-year study of 2nd and 4th graders to measure growth in their ability to use their 
visual critical thinking skills, developed through VTS, to observe art and non-art objects (2001). While measuring 
the growth of these students from their first year to their fifth year using VTS, her team also assessed the difference 
of growth from students with and without VTS treatment. Housen’s study found that students who had gone through 
VTS had more growth in their first as well as the fifth year than the students without VTS and improved 
standardized test scores. 

Philip Yenawine’s book, Visual Thinking Strategies: Using Art to Deepen Learning Across School 
Disciplines, provides a number of anecdotes and explores where VTS has been applied to other subjects, such as 
Social Studies, Science, and Language Arts. The practice of finding evidence of VTS helped students to naturally 
search for evidence to support their arguments as they complete assignments and exams. (Yenawine 2013) 

Visual Thinking Strategies has also been tested in a medical school environment. In 2010, medical and 
nursing students at the University of Texas Health Science Center San Antonio were trained to “improve their 
physical observation skills, increased tolerance for ambiguity, and increase interest in learning communication 
skills” (Klugman et al. 2011, p. 1266). Results of post-training evaluation revealed an increase in observation time 
and descriptive word count in their examination for patient images. Also noted was an increase in tolerance level 
for ambiguity and interest in learning communication (Klugman et al. 2011). These results show the effectiveness of 
VTS in non-art related courses and for older students. 

Abigail Housen and Philip Yenawine created VTS in order to increase aesthetic viewing. However, from 
research and anecdotal observations, there is evidence that VTS has been shown effective for problem-solving for 
other subjects outside of art. By implementing VTS into the undergraduate course curricula, it may be an 
opportunity for the students to improve in critical thinking skills that can be used in a variety of subjects, settings, 
and life-long learning. Thus, the study aimed to add to the knowledge base regarding VTS in relation to its 
effectiveness in undergraduate students' increasing their thinking skills. The aim of the study was to determine 
whether consistent exposure to VTS practices will increase the student's ability to utilize thinking skills. From the 
prior research conducted on K-12 and medical students, we expect an increase in observational and critical thinking 
skills. 


Methods 


Participants 

Our research subjects were students in an honors Visual and Critical Thinking course (n=18) at a large 
midwestern university. Students varied in their year in college: 33% in were first year students, 39% second year 
students, 11% third year students, and the remainder, 17% were fourth year students. Of these students, 55% were 
from college of liberal arts, 33% from the college of science and engineering, 6% from the college of business, and 
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6% from the college of biological sciences. The following majors were represented: theater arts, history, linguistics, 
finance, electrical engineering, mechanical engineering, chemical engineering, economics, sociology, physiology, 
neuroscience, computer engineering, mathematics, and art. Due to the variety of colleges represented, the knowledge 
of these students at the beginning of the course in regards to a critical examination of art is a fair representation of 
the general student population. 


Visual and Critical Thinking Course 

The instructor of the Honors Visual and Critical Thinking course held weekly VTS exercises to familiarize 
the students with the process. The weekly exercises consisted of the instructor projecting an image worked 
collaboratively to figure out what is happening within the image. The instructor used standard questions to keep the 
conversation going, paraphrased student responses, and pointed to aspects of the image the student mentioned with 
minimal interference and judgment. The instructor is a trained expert in the VTS method.. 

Week | and week 9 VTS assessments were individually completed by writing about the provided artwork 
rather than discussing as a class. For both exercises, the instructor projected and printed photo copies of the 
paintings by Edward Hopper: New York Movie (See Figure 1) in week | and Automat (See Figure 2) in week 9. In 
addition, the exercises took place in the regular course classroom, with minimal external distraction. 

Hopper's work New York Movie was used for the first assessment, and another Hopper piece, Automat, was 
used for the second assessment. Images were chosen for their similarity in content and mood. Similar paintings were 
used to lower the chance of extraneous factors influencing students’ responses. 


Fig. 1. Edward Hopper’s New York Movie (1927) Fig. 2. Edward Hopper’s Automat (1939) 


Procedure 
[1] Collected week 1 data: administered VTS Exercise to students in HSEM 2270V on the first week of the 
semester. Students were provided between 10 to 15 minutes to complete the exercise. 


[2] Completed verbal, discussion-based VTS sessions with the entire class for about 10 to 20 minutes 
during class once a week until the 9th week. 


[3] Collected week 9 data: administered VTS Exercise to students in HSEM 2270V on the 9th week of the 
semester. Students were provided between 10 to 15 minutes to complete the exercise. 


[4] Anonymized (student name and VTS Exercise administered weeks) student responses by course 
professor, Dr. Hokanson, to prevent possible interference of bias. 


[5] Measured students' week 1 and week 9 VTS Exercise using the VTS Instrument created to calculate 
VTS scores. 


[5.1] Quantified the number of separate thoughts written in the VTS Exercise response by parsing 


student responses into separate thoughts, identifying if certain thoughts are supporting observations for a concluding 
thought. 
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[5.2] Categorized each thought into the respective stage in the VTS Instrument for each student. 
[5.3] Calculate the mean count from each evaluators’ scores for each stage. 


[6] Each student’s pretest versus posttest VTS scores were analyzed using a chi-square statistic that 
compares their number of responses at each of the Aesthetic Viewing Stages. 


Research Participation 

Since this is an analysis of existing educational data, the students had a choice of whether to have their 
classwork analyzed or not by signing a consent form after completing the week 9 exercise. The consenting student’s 
data and demographics were anonymized for analysis to prevent possible bias and identification. 


VTS Instrument 

An instrument was created for use in this study by the authors to evaluate each student's observation of the 
artwork from the in-class exercise. Each thought-response was scored and categorized into one of the Aesthetic 
Stages. 


Four levels were used to categorize responses by stage. Stage | responses are basic observations within the 
image that don’t include context from the outside world. Stage 2 responses showed evidence of constructing a 
conclusion based on specific observations they had made. Stage 2 responses may also have brought in comments 
based on their interpretation of their social or moral world. Stage 3 comments show the viewer has started to step 
outside the world of the painting and wonder about the artist’s intentions. Stage 4 starts to construct meaning 
through symbolism. Because the stage 5 response is reserved for one who has gotten to know the art over a long 
period of time, Stage 5 is not applicable to this study. Stage 5 respondents are typically professionals in the field of 
art history. Expressed thoughts that could reasonably fit in to two categories were entered in both. Thoughts derived 
or developed from lower level observations were categorized at the higher level. 

Thoughts | and 3 are scored as stage 2 as they are observations that are supported by other observations. 
Thoughts 2, 4, and 5 are scored as stage | as they are observations that point that the factual and basic details. 

Since there are various ways of measuring an increase in critical thinking skills, both the overall quantity of 
responses and the quality of responses (the Stages of Aesthetic Viewing (Housen 1999)) were analyzed. 


Results 


The mean score for the number of thoughts week | stage 1 was 4.86 thoughts per participant. By week 9, the mean 
number of stage | thoughts per person increased to 5.92 amounting to almost an extra thought per person. Stage 2 
had an average of 1.44 thoughts per person in week | and 3.58 thoughts per person in week 9. Stage 3 increased by 
0.31 thoughts per person, and stage 4 only had responses in week 9 with a mean of .08. (See Table | and Table 2 for 
complete descriptive statistics) 


Table 1. Descriptive statistics of the week | scores 


n x M Mo > Ss Minimum Maximum 
wl Sl 18 4.86 4.50 4.00 87.50 1.76 2.50 10.50 
W1 S82 18 1.44 0.75 0.50 26.00 1:51 0.00 5.00 
W1 S3 18 0.08 0.00 0.00 1.50 0.19 0.00 1.50 
W1 S4 18 0.00 0.00 0.00 0.00 0.00 0.00 0.00 
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Table 2. Descriptive statistics of the Week 9 scores 


n x M Mo YX s Minimum Maximum 
W9 S1 18 5.92 6.50 6.50 106.50 2.32 2.50 10.00 
W9 S2 18 3.58 2.75 1.50 64.50 2.40 1.00 8.50 
W9 S3 18 0.39 0.00 0.00 7.00 0.58 0.00 2.00 
W9 S4 18 0.08 0.00 0.00 1.50 0.26 0.00 1.00 


Figure 4 shows the growth of each stage for the mean scores of each week, with uncertainty bars that 
represent the confidence interval at 95% for each of the mean VTS scores. Figure 5 and 6 shows each student’s 
change in their VTS scores from week | to week 9 for stage | and stage 2 respectively. 


Week 1 Mean 
@ Week 9 Mean 


VTS Score 


tN 


Stage 1 Stage 2 Stage 3 Stage 4 


Fig. 3. Growth in Mean VTS Scores 
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Figure 4. Stage 1 Individual Scores Comparing Week | and Week 9 


Week 1 
od @ Week 9 
8 
p 6 
8 
n 
n 
= 
e A 
Students 


Figure 5. Stage 2 Individual Scores Comparing Week | and Week 9 


Chi-Square Tests 
Goodness of fit: In order to determine if our data showed a normal distribution, a chi-squared test for 


goodness of fit was performed. This test looked at the number of thoughts from a particular week and stage (i.e. 
week | stage 1) and compared that number across all 18 participants. 

The results of our statistical analysis showed statistical significance in the improvement of scores from the 
VTS exercises from week 9 when compared with week 1. The scores were tallied and compiled into a matrix that 
allowed each participant to have an individual score that represented the number of thoughts they articulated in each 
of the stages, 1-4. (Developed from Abigail Housen's Stages of Aesthetic Thought (Housen 1999). Each student 
ended up with eight scores: week | stage 1, week 1 stage 2, ...through week 9 stage 4. 

An individual chi-squared test was performed for each week and stage (7 in all since there were no scores 
from week | stage 4). Then, results from stages 2, 3, and 4 were combined as a higher level category for both weeks 
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1 and 9. All tests resulted in an extremely low probability that our data is normally distributed; rather, the data seems 
to be strongly right skewed. 


Test for independence: 

It was determined that the next step should be to analyze the data using the chi-squared test for 
independence. In order to meet the minimum requirement of each cell of the chi-squared matrix, the data were 
combined for stages 2, 3, and 4. The results of the Chi-Squared test showed p =0.001499 


Discussion 


The study showed an increase in overall and higher stage thoughts between week | and week 9. This 
implies a significant increase in critical thinking skills as evaluated by the VTS instrument (p<.05). This suggests 
that the students in week 9 understood that they were expected to imply higher stage thinking while participating in 
this exercise and had the capability to do so. 

These results are similar to those of the study of medical school students. The Klugman et al. study resulted 
in an increase in the number of words used to describe the given image (2011). In both cases, participants that had 
been exposed to VTS had an increase in the number of observations made while examining an image. One would 
surmise that exposure to this type of exercises increases students! ability to look at a puzzling image and critically 
investigate details and context of an image. 


Instrument Reliability 

The VTS instrument was designed using the Stages of Aesthetic Thought (Housen 1999) outlined by one of 
the originators of VTS and a draft evaluation instrument created by the instructor, a VTS expert. Therefore, it was 
created for this study with the specific Edward Hopper paintings in mind. There are no reliability assessments for 
this instrument aside from the use of the instrument on past semester student responses. 

In addition, the authors recognize the variability potential for using this instrument. For instance, the 
qualifications for what a thought-response entails and which category each thought-response belongs are subjective 
determinations with possible bias to which each individual who scores a VTS Exercise will have differing opinions. 


Discussion of Errors 

There is very little published research on Housen’s Stages of Aesthetic Development. Our instrument has 
very little research to validate the content, which leaves chance for a systematic error. Participant’s responses are a 
product of their individual mood, external stresses, etc. when they completed the VTS Exercise. We were not able to 
select participants but have used the largest available sample. 

Our results concluded that there is a statistically significant change, the absence of a Type I error can’t be 
proven. The study was designed to reduce its likelihood. The external validity of this study was controlled by 
measuring the entirety of the class who studied VTS (n=18). The participants were of a variety of discipliens within 
the university and from all four years of undergraduate study. 


Summary and Conclusion 


This study focused on VTS that was taught in an undergraduate classroom setting at a large, midwestern 
university and measured all the students taking a visual and critical thinking course during fall semester of 2018 
(n=18). Although weekly VTS exercises were completed through week | and week 9, the class was exposed to other 
aspects of critical thinking activities within this course, which may have added to their increase in critical thinking 
skills. In addition, because the increase of knowledge and practical application of VTS was required, it can only be 
generalized to students who took or will take a similar critical thinking course that includes VTS-style learning. 
Future work in this area could involve a longitudinal study researching various years of this particular course, or 
comparing performances with other courses that do not employ VTS-style learning. 

The potential of VTS outside of museum studies is still in the early stages of investigation. Although 
studies exist on measuring skills gained through VTS (Housen 2001; Klugman et al. 2011), there is no published 
instrument for quantitatively measuring VTS, which is why we have created our own instrument for this study. 
Therefore, there is the opportunity for future work to use this VTS instrument to quantify written or verbal VTS 
exercises. The use of this instrument is not limited to similar studies but could be used on studies of experts in the 
various fields. However, it may need some modifications as the instrument was created to measure students without 
specific knowledge of art history and the work that was chosen for both exercises was a realistic style with a single 
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human figure and a recognizable context (See Figure 1 and 2). Non-subjective (abstract) art would likely give 
different responses and may not appropriately categorize in Housen’s Stages of Aesthetic development (Housen 
1999) in the same manner as the realistic paintings. 

Visual Thinking Strategies is an attractive option for learning critical thinking skills because of its potential 
for transfer to other industries. Other studies regarding VTS have measured the transfer of knowledge from art 
viewing to a specific domain (Housen 2001, Klugman et al. 2011). The participants in this study had various majors 
so there is an opportunity to study the transferability to each participant’s domain. Thus, future work in this field 
could involve a follow-up study that looks at the participant’s transfer of gained critical thinking skills as a result of 
VTS. 

This study showed a general increase from week | to week 9 in both the quantity and stage of thoughts. 
While week | results showed predominantly stage | thoughts without a notable count of stage 2, there was a great 
increase in the number of stage 2 thoughts while gaining a slight increase in stage 1 thoughts in week 9. Not only did 
the participants show that they are thinking more critically about the image in week 9, but their basic observation 
skills (Stage 1) have increased, as well. Between the two test sessions, participants' stage 1 scores increased, which 
suggests that they were still using their basic observation skills to build meaning for their higher level thoughts. 

This study supports the notion that Visual Thinking Strategies is valuable as the results showed that there 
were significant improvement in visual critical thinking scores as measured by our instrument in as little as 8 weeks. 
This finding is important as we can infer an increase in critical thinking skills if they are engaged in a critical 
thinking course that integrates VTS. 

Increasing critical thinking skills is critical for the success of modern students who are working with 
rapidly produced, evolving, and collected information. There are levels of ambiguity and unknowns in information 
found throughout the majority of disciplines and fields of study. In addition, undergraduate students are expected to 
dissect and process the knowledge that is given to them and are highly responsible for their own learning. Thus, 
other undergraduate institutions or programs may design critical thinking courses to include VTS exercises to 
develop their students’ critical thinking skills. As Yenawine and Miller stated, “Visual Thinking Strategies does not 
teach what to think, but rather supports the discoveries students make when they are given opportunities to think in 
various ways” (2014). 
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Appendix A Five Stages of Aesthetic Viewing 


The following are comments made by individuals of each stage when viewing Picasso’s Girl Before the Mirror. 
This data, collected by Abigail Housen, was used to construct the VTS instrument. The style of artwork used here is 
not of a realistic style, so modifications were made to categorize the data accordingly. 


[1] Stage I: Accountive 
“’.. [see... two women here... They... are looking at each other... looks like one of the women has a... 
misfortune...” (Housen, 1999, p. 9) 


[2] Stage II: Constructive 
“On that frame... or in that mirror there reflects some man’s face but it’s not a full face... One can see the lips the 


mouth, chin and half of the face... If you look at this man’s face... it’s the man from some other planet...” (Housen, 
1999, p. 10) 


[3] Stage HI: Classifying 
"it seems to have, the artist divides the painting into four, actually, you can also look at it in half, and it seems to be 
two different views of a woman of a female form..." (Housen, 1999, p. 10) 


[4] Stage IV: Interpretive 

"Well, the red color — is probably some aggression, and blue is, on the contrary, some feelings of rest... it's 
tensioned between these two colors... Here... changes, as she discovers something in herself... To my mind the 
point of this picture is some certain change..." (Housen, 1999, p. 11) 


[5] Stage V: Re-creative 
"I think it would be interesting to ... sit and watch Picasso do that because... you have this fantasy that it was this... 


very continuous, easy, sure, spontaneous... creation of all these forms that one flows right to the other..." (Housen, 
1999, p. 11). 
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“Representation and communication are motivated by the social; its effects are 
outcomes of the economic and the political. To think or act otherwise is to follow phantoms.” (Kress, 2005, p.6) 


Inclusivity has become a cherished value within academia, as both cultural norms and legislation prod our 
educational system to improve access for marginalized communities. Interest in accessibility is on the rise, as recent 
studies estimate that 11 percent of our students have declared at least one disability (National Center for Education 
Statistics). In the United States, the rights of students with disabilities are guaranteed through Section 504 of the 
Rehabilitation Act of 1973 as well as a patchwork of state and federal statutes aimed at improving access to 
education. 

Developing pedagogies that enhance access for all students requires that we examine our practices in the 
classroom. Many educators and instructional technologists are embracing the concept of Universal Design (Mace, 
1988) by striving for texts and teaching environments that can be of use to all students, rather than demanding that 
students declare their disabled status in order to receive specialized accommodation. The texts through which we 
communicate the substance of our courses (e.g., syllabi, readings, lecture notes) are of particular interest to the 
current research, which focuses on the reaction of able-bodied students to the introduction of EPUB, a relatively new 
and highly accessible document format. 

As Kress (2005) explained, this topic is both technological and political. Accessibility is not simply a state 
of being, it is a status that is given or denied by those who control the means of access. Printed books — for centuries, 
the primary platform for information transfer — are inaccessible to those without sight unless assistive technologies 
are made available to the affected audience. Whether such interventions are adopted easily or contentiously depends 
on the majority’s interest in providing access. Universal Design advocates a single solution that works for all; as 
such, any successful effort to improve the accessibility of our documents must first win the approval of the majority. 
In the case of EPUB versus PDF, the choice of document format is more than semantics — this discussion pits our 
traditional fixed-layout designs (also known as pages) against a modern reflowable presentation style. To elaborate 
the fundamental importance of this issue, it will be necessary to clarify a number of concepts that typically go 
unexamined. 

Our digital era compels us to leave behind notions of the book as a collection of printed paper rectangles 
bound between covers for a more essentialist idea: the book as a self-contained, portable collection of content that 
affords both presentation and navigation capabilities. The concepts of being self-contained and portable are key, 
because texts that do not satisfy those objectives (e.g., webtexts) require ubiquitous Internet access — a concept 
which disadvantages students without home Internet access, whose only off-campus connectivity may come from 
bandwidth-limited cellphone plans. Just as importantly, webtexts rely on the publisher to maintain uninterrupted 
online storage while self-contained portable documents may be downloaded and archived by the students 
themselves, guaranteeing access to key texts during future semesters. Digital documents exhibiting these properties 
are often called e-books, although their contents might amount to only a single page. 

This study’s participants interacted with two e-book formats: the ubiquitous Portable Document Format 
(PDF) and the less well-known EPUB (not an acronym). As such, it is important to explain the concerns over their 
use that inspired this research. 

Prior to the arrival of Adobe’s PDF technology, exchanging “native documents” was the status quo. This 
practice suffered from several disadvantages: the recipient needed to own the same software used by the author, 
graphics might fail to appear when the document was opened, and the recipient needed to install and activate the 
same fonts used by the author to avoid unwanted text re-wrap (due to the substituted font’s differing character 
width). PDF resolved these issues by encoding fonts and images into a single file that could be viewed and printed 
using Adobe’s free Acrobat Reader software. PDF’s success in ameliorating the exigency created by the exchange of 
native file formats resulted in its rapid — and largely unexamined — adoption within academia. Soon after, the advent 
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of screen reader software allowed the text within PDF files to be read aloud to visually impaired computer users. 
Interest in providing greater accessibility led Adobe to add an additional layer of structured data (including reading 
order instructions as well as alternate descriptions for images) to the PDF format. These accessibility features, 
however, are not required components and are separate from the PostScript-based object list that creates the PDF’s 
visual representation. Additionally, the process of creating a fully accessible PDF is very complex, is typically only 
partially completed by authors, and requires the purchase of Adobe’s full Acrobat Pro software application. 

Due to these concerns, users of assistive technologies cannot be certain as to whether a newly encountered 
PDF document will be readable, or whether accessibility features will be properly implemented. As a result, 
assistive technology users may encounter difficulties accessing PDF content. As noted by academic journal editor 
Trude Eikebrokk (2014, para. 7), “PDF is a format that can cause many barriers, especially for users of screen 
readers (synthetic speech or Braille).” 

The origins of the EPUB format preceded the introduction of Adobe’s PDF. In 1993, the Swedish Library 
of Talking Books and Braille commissioned a commercial software company (Labyrinten Data AB) to develop the 
Digital Audio-based Information System (DAISY) “talking book” or DTBook, a forerunner of today’s accessible 
EPUB format. This differs from PDF in numerous substantive ways; most notably in that they are easily parsed 
collections of XHTML, XML, and CSS files. The simplicity of these markup languages stands apart from PDF’s 
mixture of ACSII and binary data (Lukan, 2012), which can only be displayed by applications that support the 
complex Adobe Imaging Model (Adobe, 2000). 

DTBooks were specifically referenced by Rose et al. (2006) in their initial treatise of Universal Design for 
Learning (UDL) as a format for students with “print disabilities” (Wise, 2016), including dyslexia and other 
cognitive disorders in addition to various levels of visual impairment. DTBooks were intended to support UDL’s 
requirement for multiple means of presentation to accommodate the various ways that students interact with and 
learn from courseware (Rose, 2006, p. 136). Unfortunately, DTBooks are not actually a good fit with the concept of 
Universal Design, as DTBooks (and the reading systems used to peruse them) are not intended for use by the sighted 
majority. 

Recently, the DAISY Consortium and many other accessibility advocates have suggested the adoption of 
EPUB3, a recent iteration of the EPUB e-book format, as the universally accessible replacement for DTBooks 
(DAISY Consortium, 2011) but their voices are struggling to be heard in a world dominated by commercial 
purveyors of “accessible PDF.” Progress is evident, however, in the growing number of academic databases that 
contain EPUBs (e.g., ABC-CLIO, ACLS Humanities, EBSCO E-Books, Proquest’s Ebrary, Elsevier’s 
ScienceDirect, Taylor & Francis), and the many sessions about EPUB found at academic conferences. 

The current research responds to a gap in the literature on UDL, in that this heralded learning theory has 
not yet explicitly adopted DTBook’s specified replacement, EPUB3 (DAISY Consortium, 2011). The need to 
research UDL practices that involve EPUB3 is clear, in that the EPUB3 format is more applicable than DTBook to 
the underlying premise of Universal Design: that we should avoid the provisioning of alternate accommodations for 
those who differ from the majority, but instead strive for universally acceptable formats “which, to the greatest 
extent possible, can be used by everyone” (Mace, 1988, p. 3). 

By observing a small number of abled college undergraduates as they interact with EPUB documents and 
speaking to them at length about that experience, this qualitative research hopes to add to the discussion of whether 
“typical” college students can be successfully motivated to adopt this new technology. Specifically, my purpose was 
to better understand the potential for user resistance to EPUB as a replacement for “accessible PDFs” in academic 
settings, with a special interest in the role that altruism might play as a motivating factor in EPUB’s adoption. 


Review of Relevant Literature 

Society’s essentially unexamined embrace of PDF for document distribution is evidenced by the lack of 
journal articles on the ramifications of its adoption or its alternatives. In response, this study presents an examination 
of empirical literature that address motivation’s role in technology adoption and the value of altruism as one facet of 
motivation, as well as the invisible nature of non-apparent disabilities, and what appears to be the only previous 
peer-reviewed research study on student preference for EPUBs. 


Motivation and Altruism in Technology Adoption 

Altruism was not considered to be an important component of motivation in the seminal work of Vroom 
(1964). In keeping with the behaviorist thinking of the period, Vroom’s expectancy-theoretic model of “motivational 
force” depicted human behavior as a simplistic effort to gain pleasure while avoiding pain. In his oft-cited research 
on word-of-mouth communication, however, Sundaram et al. (1998) included altruism as a contributor to 
motivation, specifically defining it as “the act of doing something for others without anticipating any reward in 
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return” (p. 529). This concept is key to understanding the inclination of abled readers to adopt the EPUB format in 
light of its greater accessibility for disabled users. 

One established way to examine users’ willingness to transition to a new technology is the Technology 
Acceptance Model (TAM) developed by Davis (1989). Chen et al. (2017) extended Davis’ TAM to include the 
additional motivational factors of social interaction, enjoyment, and altruism. Chen et al. found altruism to be highly 
significant component of social interaction (p. 8) even though the nature of these interactions was virtual rather than 
face-to-face. Chen et al.’s research supports a key assertion of this study: that interest in helping others can be a 
motivational force for technology users.Hernandez et al. (2011) reported on the significance of altruism in 
determining attitude towards and usage of Information and Communication Technology (ICT) tools. Their research 
found that participants in online courses were significantly motivated by altruistic feelings toward their classmates, 
even though the virtual nature of these courses diminished the likelihood that their altruistic behavior would be 
reciprocated (p. 2228). Hernandez also determined that students were inherently interested in obtaining the approval 
of the course instructor (p. 2230). 

The document formats examined in this research have been included in electronic knowledge repositories 
(e.g., university library portals, Blackboard’s eReserves feature). Kankanhalli et al. (2005) investigated the 
motivation of knowledge management practitioners and found that the intrinsic benefit of altruism was strongly 
correlated with the motivation to share knowledge (p. 131). 


Disability Non-disclosure 

The principal tenet of Universal Design (that the same version of a thing should be usable by everyone, 
regardless of their particular abilities) becomes even more important when we consider how difficult it can be to 
identify which students are in need of accommodation. 

Kranke et al. (2013) found that “students with non-apparent disabilities encountered stigma from peers and 
professors” (p. 36), which creates pressure for students to maintain their undisclosed status whenever possible. 
Students identified a desire for autonomy and normality as additional reasons for avoiding disclosure (p. 43), as well 
as concern about maintaining compliance with their professor’s instructions so as to avoid undermining the 
professor’s view of their capabilities (p. 43). 

Salzer et al. (2008) conducted a survey of students diagnosed with mental illness. A majority (56%) said 
they felt embarrassed when alerting professors to their disability and were fearful that these professors would 
stigmatize them (p. 373). Forty-two percent of students who had disclosed their condition characterized their 
professors as being unreceptive or even uncooperative (p. 373). These findings support the position that many non- 
apparent disabilities go unreported, strengthing the case for universally accessible texts as a way to obviate the need 
for student self-reporting. 


Digital texts in the classroom 

Literature on the use of self-contained, portable electronic document formats in the classroom (as opposed 
to HTML-based presentations) is in short supply; studies that compare multiple forms of electronic documents are 
rare. Standalone, archivable document formats, however, deserve in-depth analysis because the format that emerges 
as our final choice will become the accepted replacement for printed class texts. 

In what may be the only peer-reviewed study of its kind, Mills (2016) conducted a quasi-experimental test 
of student preference between a course textbook distributed in one of three e-book formats: PDF, KF8 (Amazon 
Kindle documents), and EPUB. Some students received an EPUB containing interactive content, while others 
received either a PDF file or a KF8 (Kindle) file. Participants who received the EPUB file reported significantly 
higher levels of interaction and engagement with the course text, as well as greater perceptions of the text’s 
usefulness and value (p. 130). These findings support the current research’s recommendation of EPUB as a format 
well-suited for sighted readers as well as users of assistive technologies. 

Reviewing this published literature provided an understanding of several relevant issues. Despite being 
overlooked in early work on the topic, altruism has been empirically shown to affect motivation and knowledge 
sharing; as a result, any modern investigations of technology acceptance should include altruism among their 
motivating factors. This review also lends credence to Universal Design for Learning’s recommendation of texts that 
are compatible with assistive technologies, since the non-disclosure of psychiatric and attentional disabilities is 
problematically low, contributing to a large number of students who avoid requesting accommodations despite 
awareness of their availability. We also see that while much more research needs to be done in the area of student 
digital textbook preference, at least one prior study has identified a student preference for EPUBs over PDFs or 
Kindle books. While each piece of literature reviewed offered some valuable contextualization of the issues at hand, 
none specifically addressed the practical question at the heart of this research: if we recommend EPUB3 in response 
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to Rose’s call for the use of highly accessible document formats in our classrooms, can abled students be motivated 
to adopt this new technology? 


Methods 

The purpose of this research is to investigate students’ attitudes toward a new instructional technology 
(EPUB) as a replacement for “accessible PDFs” in academic settings. To further this goal, I conducted qualitative 
research to discover the actions and opinions generated when students without print disabilities were asked to 
interact with one document saved as both a PDF and an EPUB. These observed activities, as well as the semi- 
structured questions that both preceded and followed the hands-on portion of the interview, were designed to cast 
light on these research questions: 
RQI: What is participants’ knowledge of the EPUB format? 
RQ2: What challenges are reported or observed when students are learning to access EPUBs? 
RQ3: How do participants perceive EPUB’s constraints and affordances? 
RQ4: According to participants, does a discussion of Universal Design principles increase their interest in the EPUB 
format? 


Research setting 

This research took place at a former teacher’s college located in a small Midwestern city (population 
43,849), located approximately one hour’s drive west of Chicago. The university’s Fall 2018 enrollment was 
comprised of 17,169 students (including 1,211 international students), 75% of whom are undergraduates. Males 
make up 50.8% of the undergraduates, while 49.2% are female. The average undergraduate student age is 22. Fifty- 
five percent of these undergraduates self-identified as White, 15.8% as Black, 17.9% as Hispanic/Latino, and 5.4% 
as Asian. As a university with a significant international enrollment located near America’s third most populous 
city, it is considerably more diverse than the county in which it is located, 1.e., 86.6% White, 11.4% Latino, 8.1% 
Black, 2.8% Asian (U.S. Census Bureau). 


Participant demographics 

Six undergraduate students (ages ranging from late teens through early 20s) from this Midwestern 
university were interviewed; the sample consisted of an even split between male and female participants; 50% were 
Black (one female, two male), and 50% were White (two females and one male). Each student was enrolled in a 
different major: Accounting, Communications, Kinesiology, Nursing, Psychology, and Special Education. The 
participants comprised a convenience sample drawn from my former students from an on-line class; 12 students 
were invited to participate via email, but only six responded affirmatively (50% response rate). Students’ 
participation was incentivized through an offer of $15 cash, which was given to the interviewee at the start of each 
session. 


Data collection and analysis 

Data was gathered during individual face-to-face interviews; each interview lasted between 28 and 45 
minutes. The interview site was a small conference room on campus. The participants were asked to examine the 
same document in two different formats (PDF and EPUB) on whatever computing platform they preferred; half of 
them (three) used a laptop, while the other half (three) used a smartphone. Among the laptop users, two were 
running Microsoft’s Windows operating system while one was running Apple’s MacOS. All three smartphone users 
interacted with an Apple iPhone. While using these devices, the participants were asked to interact with the two 
documents in various ways: scrolling, reading, searching, navigating to a specific section, and enlarging the 
document’s text. 

These interviews were recorded with a high-definition video camera placed behind the students’ right 
shoulder in order to capture their hand movements as they interacted with an EPUB on their device of choice. These 
video recordings allowed me to replay and closely observe the students’ interactions with the hardware and software 
used to access the documents, in order to note any challenges or general patterns that emerged from their use of 
these familiar and unfamiliar document formats. 

Audio from these recordings was transcribed via F5 Transcription Pro software for open coding in Nvivo 
12 for Mac. An initial pass through all six transcriptions produced 24 codes; a second pass caused me to merge two 
codes then organize thirteen others under six top-level codes. This resulted in a total of 16 top level codes from 
which to generate themes. Several of these were related to aspects of motivation (altruism, compliance, efficacy, 
convenience) while others applied to the hands-on tasks performed by participants during the interview (e.g., 
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controlling text size, navigation, search/find) or their own statements on various related concepts (e.g., awareness of 
EPUB and Universal Design, general challenges with reading texts, preferred study locations). 


Researcher role 

The class during which I previously became acquainted with these students was conducted online rather 
than face-to-face, so our relationship remained relatively impersonal. My role as their former instructor and our 
substantial age differences made them likely to perceive me as an “outsider,” although our shared experience of the 
online course might also have granted me a small measure of “insider” status. As Dmitriadis (2001) observed, the 
researcher is always some mix of insider and outsider, and so I endeavored to keep both attributes in the proper 
perspective. Acknowledging and accounting for such concerns is an essential task for researchers, according to 
Dmitriadis, since these multiple versions of our identities “often work at distinct cross purposes and can inextricably 
complicate and even derail our research as we originally conceive it” (p. 579). 


Concerns 

My “insider” role (garnered through my identification as a member of the study group) seemed unlikely to 
pose a substantial problem due to the impersonal nature of our prior distance-learning relationship, but a larger 
concem came from our previous shared experience with the topic of EPUBs. These students were selected 
specifically because I knew they had some small familiarity with the idea of an EPUB file as a self-contained 
portable document format. They gained this knowledge by participating in an extra-credit assignment as students in 
my online class; each had earned a small improvement to a previous assignment grade by downloading one short 
EPUB file from our class website (on Blackboard), then taking a short quiz about their interaction with the 
document. 

Just as these participants came to their interviews with a small amount of pre-existing familiarity with this 
topic, I too brought my own prior knowledge to our sessions. Having interacted with more than a half-dozen 
sections of undergraduates during prior teaching experiences, I held some assumptions about their existing skill 
level when it came to digital technology. Upon reflection, I determined that the participation of these students in the 
current research was appropriate, as their past experience would reduce the initial strangeness of the EPUB format, 
thereby allowing for a more accurate appraisal of their abilities to interact with it. Additionally, it was undoubtedly 
clear to these participants that I am an advocate for the use of EPUBs within academia. However, at the point when 
this research occurred, they were no longer my students and had no compelling reason to modify their behavior in 
order to gain my favor. In addition, I feel that self-reflections on this subject have allowed me to approach this 
research impartially, as I have remained cognizant of the potential for bias. Such self-reflective writing on the basis 
for potential bias is one technique recommended by Lawrence-Lightfoot and Hoffmann Davis (1997), who noted 
that “making the anticipatory schema explicit (in the form of memos, journals, or self-reflective essays) allows for 
greater openness of mind” (p. 186). 


Findings 

The current research was conducted to collect student perspectives on the adoption of a new instructional 
technology, i.e., EPUB. My initial expectation was that participants might speak of altruism’s connection to 
technology adoption, in addition to such classical motivations as learning goals and performance goals (Dweck & 
Leggett, 1988). The statements and observed actions of the participants in this study caused me to rethink those 
initial expectations, as EPUB’s benefits for sighted users came to eclipse the moral advantages of accessibility. The 
findings that resulted from these interactions have been organized thematically as responses to my research 
questions. Pseudonyms have been used to protect the participants’ identities. 


EPUB remains largely unknown to students 

Although these participants shared a past exposure to the existence of EPUB documents gained during an 
optional extra-credit assignment in the prior semester, they were unanimous in saying that they had not heard of the 
EPUB format prior to the extra-credit assignment, nor had they heard the term used since that initial exposure. 
However, two of the participants (Emma and Francine) stated that they were using Apple’s iBooks software on their 
laptops, while Bernadette said that she sometimes used the iBooks app on her smartphone. This indicates an ongoing 
challenge for those who strive for greater awareness of the EPUB format within academia: software vendors have 
successfully branded the EPUB books sold on their e-book marketplaces with the name of their particular 
applications. EPUBs purchased through Apple’s iBookstore are referred to as “iBooks,” while EPUBs purchased 
through Barnes & Noble’s marketplace for use with their Nook e-readers are called “NookBooks.” Even some 
academic publishers contribute to this lack of name awareness — the University of Chicago Press sends out a 
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monthly email offering a “free e-book,” but subscribers never see the term EPUB in either the initial email or the 
subsequent HTML landing page. 

These branding issues result in a real problem for those who would encourage broader adoption of the 
EPUB format. Emma, owner of both an Apple laptop and an iPad, said, “I've used iBooks in the past, and I was 
aware that [the ability to change text size] was available.” Her ability to modify font size when reading on her 
devices (rather than zoom in and out, as with a PDF) indicates that she must have had prior experience reading 
EPUBs — yet when asked earlier in the interview whether she had ever heard of EPUBs after the prior semester’s 
extra-credit assignment, she said, “No.” It will be difficult to build consensus around the advantages of the EPUB 
format unless users are able to identify this format by name. 


Users report minimal challenges during EPUB adoption 

In order to participate in the extra-credit assignment previously mentioned, each of the participants had 
already gone through the process of assuring that they could open and read an EPUB file. For three of the 
participants, this required no additional effort as they were already using Apple devices that ship with the pre- 
installed EPUB reader iBooks. Another participant also used pre-installed software to read his EPUB: Dominick, 
whose recently purchased Windows laptop included Microsoft’s Edge browser, which can natively open EPUBs 
without a plug-in. The remaining Windows-using students reported that installation of the necessary free software to 
read EPUBs was unproblematic. When I asked Charles whether he thought that other students would find it too 
difficult to install an EPUB reader onto their preferred reading platform (e.g., laptop, desktop computer, tablet, 
smartphone), he responded: “I'm pretty sure students are used to being handed something new. Downloading it on 
their phone and getting used to it.” However, a note of caution on this topic was sounded by Anton, who said: 
A lot of people are very lazy, don’t like to do work anymore. They like a lot of work done for them the easy way and 
stuff like that, so I feel like it would be a complaint if stuff were harder and they had to put in more work. They 
would definitely complain. 
This indicates the need for students to be motivated to download and use EPUB readers as part of their coursework, 
a topic that will be explored next. 


Motivation through perceived efficacy and convenience 

Videotaped observations of users interacting with both PDFs and EPUBs supported their overwhelmingly 
positive statements regarding their experiences accessing EPUBs, but also yielded some unexpected results. It was 
surprising to discover these students’ minimal knowledge of methods for interacting with PDF documents. Each 
denied that they had ever used bookmarks to navigate their way through a PDF, or even used the search function to 
locate a specific term within a PDF; similarly, none were familiar with the highlighting tool built-in to most PDF 
readers. As a result, students were excited to experience what they perceived to be the greater efficacy of the EPUB 
format with regard to accomplishing several standard tasks associated with reading course texts: navigation, 
search/find, and adjusting text size for readability. 

The following descriptions of user interactions with EPUB-reading software refer to the two types of Table 
of Contents (TOC) often found in an EPUB, which I will explain here. One of the ways in which EPUB accessibility 
is superior to that of PDF is the requirement for EPUBs to make their contents navigable through any reading 
system’s built-in Table of Contents menu (a feature not found in PDF reading systems, even when “accessible 
PDFs” are used). Often called the hardware TOC, this navigation system is immediately available in a consistent 
fashion from any page of any EPUB. Alternatively, EPUBs may (or may not) also contain an in-book TOC, 
analogous to the Table of Contents page in a printed book. As with a PDF file that contains an in-book TOC, the 
chapters and sections listed may (or may not) be shown as clickable hyperlinks. The example files provided for the 
hands-on component of these interviews reflected the typical “real world” state of these formats, in that the EPUB 
contained a hyperlinked in-book TOC while the PDF did not. 

Navigation is made easier by consistent access to a Table of Contents. When asked to navigate to the 
"third section" of an EPUB viewed in Adobe Digital Editions (ADE) for Windows, Anton said he would scroll. 
Upon refamiliarizing himself with the fact that EPUBs change pages via a horizontal page-flipping convention 
rather than the vertical scrolling technique used by most PDF readers, he went to the EPUB’s third page. Once I 
pointed out that he was not at the third section but rather the third page, he moved backwards through the pages until 
he found the in-book Table of Contents then clicked on the appropriate hyperlink for the second section. Similarly, 
Dominick also used the in-book TOC hyperlinks to navigate to the third section when he viewed an EPUB with 
Microsoft’s Edge web browser. When asked to locate the third section of an EPUB within the iBooks app on an 
iPhone, Francine initially swiped her way through two pages before deciding (without any prompting) to instead 
click on the icon for iBook’s hardware TOC, allowing her to very quickly reach the third section. The remaining 
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users (Bernadette, Charles, and Emma) all selected the hardware TOC immediately upon being requested to navigate 
to the third section. Francine’s explanation of her behavior was typical of all the participants who accessed the 
hardware TOC: “If you want to find a page, you just use this scroll thing, but yeah, if I wanted to find a chapter, I 
would just go to the Contents thing.” Suspecting that the simple interface of an EPUB reader made it easy for the 
participants to recognize standard icon conventions (such as a menu of contents), I engaged Charles in this dialog: 
Interviewer: So show me how it was that you got to it. What was the method that you used? [Subject shows the 
built-in TOC menu.] Oh, so you went right into the navigation system! 

Charles: Yeah. 

Interviewer: And so even though you'd never touched this phone before, or used that software before, it was 
intuitive that you could kind of tell that that thing up in the corner was an Index. 

Charles: Yeah, it provides like a dropdown list or something. 

Interviewer: Yeah, nice. Now are you familiar with seeing something like that in PDFs, or is there any kind of 
Index that you use when you’re going through PDFs? 

Charles: Mmm, not that I can remember. 

The interviewees’ ability to easily locate the hardware TOC within two different EPUB readers (Adobe Digital 
Editions and Apple iBooks) on three different platforms (a Windows laptop, a Mac laptop, and an iPhone) shows the 
value of a streamlined user experience that follows consistent user interface conventions. 

Search capabilities are underutilized by PDF readers. Surprisingly, all participants denied having 
knowledge of how to search for a specific term within a PDF file. Several knew the keyboard shortcut for search 
(control-F on a PC, command-F on a Mac), but only associated that process with web browsers. Each participant 
found it easy to search for a specified term with an EPUB. Dominick, the student who viewed his EPUB within the 
Microsoft Edge web browser, used the keyboard shortcut control-F to bring up the search function; the remaining 
interviewees were all able to quickly identify the magnifying glass icon that represented the EPUB reader’s search 
function. As an example, when asked if she had any idea how to search for a term within an EPUB, Emma 
responded, “The magnifying glass at the upper right-hand corner.” 

Conducting a search within a typical PDF viewing application takes the reader to the first highlighted use 
of that term within the document; if that location is not the desired spot, the user clicks a forward-arrow icon or 
repeats the keyboard shortcut to be taken to the next highlighted instance of the term. All the EPUB readers used by 
these participants, however, follow a different convention: conducting a search pops open a floating list of all 
locations of the term, where each instance includes a snippet of the text surrounding the term (to provide context). 
The interviewees found that method to be more efficacious. When asked if the search procedure for EPUBs seemed 
to be the same as she was used to from using PDFs, she said: “No, I feel like this is different. Like you said, usually 
they just kind of give you the word and then where it is.” 

Search functions are a key capability for any researcher, but these undergraduates were unfamiliar with 
how to perform searches in PDFs. This may be a result of teachers’ common practice of providing texts to students 
as unsearchable scanned images, rather than as searchable text. If course materials are not consistently searchable, it 
is understandable that students would not learn to associate such functionality with the PDF format. 

Enlarging text for readability is the key to usability on smartphones. While the laptop users in this 
study were not overly concerned with text size, the subjects who used a smartphone to access the texts were 
immediately challenged by the tiny size of text within PDFs. Since the default behavior is for PDF readers to display 
the full width of the PDF, the text was rendered too small for students to read. Rotating the phone to landscape mode 
enlarged the PDF to fill the width of the screen; as Charles noted: “I could read it if I focused, but that's going to 
strain my eyes, so yeah, to turn it sideways would be better.”” However, the interviewees found it uncomfortable to 
navigate the PDF when only a small portion of the page was visible, causing them to flip the phone back and forth 
between portrait and landscape orientation during use. 

When the smartphone users examined an EPUB on their mobile devices, they were immediately impressed 
by the difference in readability. EPUB readers feature reflowable text that is not bound by a particular page 
geometry, so the default text size was immediately readable. The participants also found it easy and intuitive to 
change the size of the displayed text by clicking the icon representing text size: two adjacent instances of a capital 
letter A, shown at different sizes. For the three participants who were using laptops, I demonstrated the default text 
size on my iPhone. All the interviewees then began to conjecture about how this capability would increase the 
likelihood that they would use their smartphones to occasionally read assigned texts, especially during short breaks 
between classes or while traveling. The convenience of being able to use a smartphone to read class texts was noted 
by all participants. As an example, Bernadette said: “With it being on your phone, you can change the font size to 
however large or small you need it. So, you can do the reading on the go, so you can get it done.” Similarly, 
Dominick said: “TI feel like EPUB on a cellphone would probably be a lot better than a PDF because once again, like 


108 


I said, it's all blocks, blocks of text.” (Dominick had complained that the vertically scrolling display of PDFs on his 
cellphone made them appear as dense blocks of text rather than paragraphs.) Charles specifically called out the 
convenience of EPUB’s usability on his mobile device: “I look for convenience. So I usually, if I can read it on my 
phone and get the job done, then I will.” Emma noted that she’s likely to have much more opportunity to read course 
texts on her phone when away from home when she said pointedly, “You never forget your phone!” 


Universal Design as a motivational factor for EPUB adoption 

The current research found that students were more likely to support EPUB adoption after learning that the 
format’s accessibility is superior to that of PDF. In other words, the resistance of able-bodied students towards any 
additional effort needed to implement EPUB (i.e., to learn new procedures, and the potential need to download and 
install free software) could be minimized by an altruistic desire to participate in an accessible Universal Design 
pedagogy. It is important to note that none of the interviewees who participated in this research self-identified as 
having a learning disability, and that of these six undergraduates, only two were marginally familiar with the term 
Universal Design. 

Altruism provides only limited motivation for technology adoption. Anton was among the majority 
who were unfamiliar with the concept, but he soon revealed that he was familiar with the inclusive practices enabled 
by Universal Design, noting that “We had, like, wheelchairs in our class, kids with disabilities in our class, so now 
it’s like normal for us. Our generation, we’re like used to it.” He also saw the value of materials and practices that 
enabled inclusive learning and supported the idea that accessible materials would also be valued by his classmates: 
“Tt’?'d make everybody want to use it [EPUB] more, since they’d know like okay, this isn’t just for me, it’s for other 
people too with disabilities and stuff like that.” However, when asked how much additional difficulty the typical 
college undergraduate might tolerate when being asked to adopt a new accessible technology, Anton once again 
turned skeptical. “If it’s more work, yeah, people will complain,” he said, “I know my generation, a lot of people are 
lazy.” 

The two students who were already familiar with Universal Design supported Anton’s view that the 
majority’s largesse could not be taken for granted. Francine, a Psychology student who said that she was broadly 
familiar with the term Universal Design but had no knowledge of its applications, noted, “I wouldn’t be more 
motivated to use [the EPUB format], because I wouldn’t need it, but for other reasons I would be, like the things I 
said earlier that attracted me to the EPUBs. [She enjoyed the ability to change font sizes or switch into Night Vision 
mode, as well as swiping through documents.] But unfortunately, that Universal Design thing wouldn’t motivate me 
personally to use it more.” Like Anton, she felt that touting the direct benefits to users would be key in easing the 
adoption path for EPUB. In contrast to Anton’s skepticism, though, Francine thought that there would be little 
resistance if teachers mandated the use of EPUB. “Obviously it would be beneficial to people that need it, whether 
or not I need it; so, I wouldn’t care, personally,” she said. “It was a small hurdle. No difficulty for me at all.” 

When interviewees revealed a deeper personal connection to issues of accessibility, however, Universal 
Design was identified as being more motivational. Emma, a Special Education major who volunteered that she has 
“thought about this a lot,” said that “The issue isn't whether or not it's fair, it's making sure everyone has the same 
learning abilities.” Her responses were the most specific in this regard, but all the participants expressed some level 
of support for mainstreaming students with special needs into the general classroom population whenever possible. 

Personal connection to disability strengthens motivation for Universal Design adoption. Given that all 
participants saw the Universal Design component of EPUB as an advantage, it’s logical to presume that a personal 
connection to this topic would lead to a stronger connection and further reduce resistance to implementation. 
Bernadette, a second-year Nursing student, revealed that her familial relationship with disability has resulted in 
greater appreciation for the concept of Universal Design: “I think that helps make people more receptive to the idea 
[of using EPUB]. I come from a home with two brothers who are mentally impaired, so I know it sparks my 
interest,” she revealed. “Because I know my brothers learn at different rates and one likes to read, the other doesn't 
like to read but likes to listen to things, so I think it would be, like, perfect.” Bernadette’s response validates an 
approach in which instructors first make the case for EPUB adoption as an accessible format that can benefit all 
students before pointing out the ways this new format can benefit the sighted, smartphone-wielding majority. 


Compliance 

One emergent finding that was not anticipated by this study’s research questions is the strong role of 
compliance on student motivation. As noted by Kranke et al. (2013), students are highly concerned with maintaining 
a favorable relationship with their instructors — or as Vroom (1964) might have positioned it, to avoid the pain of 
low grades. Emma echoed a sentiment expressed in various degrees by all the other interviewees — that students 
would simply accept the transition to a new technology (EPUB) if instructed to do so by their teacher: 
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If I were to sit down, I came to NIU, I'm a freshman and my teacher — my professor — hands me an EPUB, well 
that’s how I’m going to learn to read and I'm not going to question it. People are just going to get used to it and it's 
going to become the normal. Because it does have those features for people with special needs or learning 
disabilities and if they need them they can have them. So, I don't think it's inconveniencing anyone, and I don't think 
many students would question it. 

Some scholars might challenge Emma’s statement by citing Activity Theory (Engestrom, 2000); under that 
premise, students who had operationalized the use of PDF would suffer discomfort when their proximal zone of 
development was de-centered by the introduction of EPUB (Vygotsky, 2011, p. 203). This research, however, 
reveals that these twenty-first century scholars have become accustomed to constant change in their selection and 
use of technological tools. Anton expressed confidence in his classmates’ ability to implement the new software 
needed to read EPUBs, saying that “Everybody’s technology savvy these days; everybody knows technology, so it’s 
not really that hard to figure out.” 

Emma extended Francine’s “no big deal” assessment of EPUB adoption by positioning the introduction of 
new technologies and formats as the duty of the instructor: “I think it's important, especially with technology 
nowadays, that we're exposing students to that. Not to say textbooks aren't functional, but jobs nowadays will never 
hand you, ‘Here's your [printed] manual.’ They’re gonna say, ‘I’m going to e-mail you that manual.’” 

From Emma’s point of view, the medium is clearly as important as the message; content wrapped in the 
outmoded trappings of print culture is seen as having less real-world relevance to today’s technology-driven 
students. 


Conclusions and Implications 


This study examined the role of student motivation in the adoption of a new instructional technology, 
EPUB. An analysis of participants’ statements and actions during the adoption process led to the identification of 
four motivational factors: compliance, efficacy, convenience, and altruism. 

One unexpected result was that participants placed the greatest emphasis on compliance, as they expressed 
the desire to use whatever tools were recommended by their professors. As initially assumed, they were also highly 
interested in the efficacy of EPUB; observation showed that they quickly learned to navigate and search an EPUB’s 
contents as well as adjust its text for optimal viewing on a mobile device. Surprisingly, there was lower (albeit still 
positive) interest in the other two factors: convenience (i.e., smartphone compatibility) and altruism (vis-d-vis 
EPUB’s enhanced accessibility). The altruistic response was more pronounced among students who possessed a 
personal connection to disability. 

Participants expressed positive views of all four motivational factors, so this implies a low potential for 
resistance when EPUB is introduced into the classroom. These findings validate an approach in which instructors 
present EPUB as an accessible format that can benefit all students, especially users of assistive technologies. Since 
the EPUB format was developed by advocates for the visually impaired, it is highly compatible with assistive 
technologies, thereby reducing the need for students to seek special accommodation. Once instructors introduce the 
concept of accessibility, they should herald EPUB as a format that is easy to use and smartphone-friendly. 

These results have significance for the growing community of educator who have restructured their 
curriculums around the concept of Universal Design for Learning, as the EPUB format enables a universal approach 
to the dissemination of course documents. Beyond practitioners of UDL, the EPUB format should be embraced by 
all educators desirous of truly inclusive pedagogies. The current research also has implications for the broader 
subject of student technology adoption, as it supports Chen et al.’s (2017) extension of Davis’ (1989) Technology 
Acceptance Model to include altruism. Additionally, it supports Kranke et al.’s (2013) conclusion that compliance is 
a powerful motivator of student behavior. Such results can inform research into the adoption of other formats and 
technologies beyond EPUB. 

While this small-scale study featured only a handful of participants in an interview setting, large-scale 
experimental studies should be conducted so that researchers can extrapolate their findings to a more 
generalized student population. Additional qualitative research on this subject is also needed to expand the 
pool of available data on student technology preferences. 
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Abstract 


The study examined the effect of a peer-teaching practice with synchronous technologies on preparing 
teacher candidates to teach digital citizenship. Candidates were required to work in small groups to 1) design, 
develop and deliver a 25-30 minute online workshop to their peers and 2) attend their peers’ workshops. A mixed- 
method design was employed to collect data including a survey and a systematic examination of rubric responses 
and workshop video archives. The results showed that the practice helped prepare teacher candidates to teach digital 
citizenship. Candidates learned new knowledge, strategies, and resources from the practice and recognized its 
helpfulness to their learning about digital citizenship. 


Introduction 


Due to technology advances, younger generations tend to have more knowledge about digital technologies 
than many adults do (Kara, 2018). However, knowing more about technologies does not guarantee that they know 
how to use technologies in an appropriate way, which causes problems not in compliant with digital citizenship 
standards (e.g., violation of intellectual property rights). Digital citizenship is defined as “the practice of defining the 
norms of appropriate, responsible behavior with regard to technology use” (Dotterer, Hedges, & Parker, 2016, p.59). 
Ribble (2015) proposed nine elements of digital citizenship, including digital access, digital commerce, digital 
communication, digital literacy, digital etiquette, digital law, digital rights and responsibilities, digital health and 
wellness, and digital security. Teachers, administrators, and parents have the responsibility of training younger 
generations to become good digital citizens and ensure they receive comprehensive instruction in digital citizenship 
(Dotterer et al., 2016; Isman & Canan Gungoren, 2014). 

How to prepare teacher candidates to achieve competency in digital citizenship so that they can better 
educate their own students in the classroom is a topic worthy of discussion. Faculty members in teacher preparation 
programs have to offer training opportunities to help teacher candidates transfer what they have learned within the 
programs to real classroom settings (Faulkner-Beitzel, 2008). As more and more universities are offering online 
programs related to instructional technology for teacher candidates, it is becoming more important to research best 
practices to help teacher candidates gain competency in digital citizenship in online learning environments. 

Peer-teaching promotes active learning and requires teacher candidates to take ownership of their own 
learning (Stigmar, 2016). It allows candidates to play different roles in the teaching-learning conversation and offers 
opportunities to achieve meaningful conversations with their peers (Garbett, 2011). There are formal and informal 
forms of peer-teaching. Formal forms include peer-tutoring, presentation or group work; informal forms include 
peer collaboration outside the class (Priharjo & Hoy, 2011). According to Vygotsky’s Zone of Proximal 
Development, candidates can learn better with the scaffolds from more knowledgeable others (Haider & Yasmin, 
2015; Vygotsky, 1978). 

In an online program, using synchronous technologies gives teacher candidates opportunities to interact 
with a faculty member or peers in real time, just as they would in the traditional classroom (Brown, Schroeder, & 
Eaton, 2016; Chen, Ko, & Kinshuk, 2005). Advantages of using synchronous technologies include 1) offering 
immediate two-way feedback (Grogan, 2015), 2) reducing the feeling of isolation (Elluminate, Inc., 2009), and 3) 
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enhancing a sense of community (Elluminate, Inc., 2009; Parker & Martin, 2010). A synchronous communication 
tool can include features such as video, audio, text chat, instant polling, whiteboard, application/desktop sharing, 
emoticons, and breakout rooms that promote collaboration among instructor and learners (Brown et al., 2016; 
Martin & Parker, 2014). The commonly used synchronous tools include Adobe Connect, Blackboard Collaborate, 
Cisco WebEx, Google Hangouts, GoToMeeting, Saba Centra, and Skype. 


The Current Study 


The purpose of the current study was to examine the effect of an online synchronous peer-teaching practice 
on preparing teacher candidates to teach digital citizenship in their own classroom. The online synchronous peer- 
teaching practice here referred to an online professional development workshop using Blackboard Collaborate. 
These two terms were used interchangeably in this paper. The following research questions were used to guide the 
study: 

1) How did teacher candidates perceive the effect of the online synchronous peer-teaching practice on 
preparing them to teach digital citizenship in their own classroom? 

2) What did teacher candidates learn in the peer-teaching practice from acting as both a workshop 
instructor and a learner? Did they see things differently when playing these two different roles? 

3) Were there any changes to teacher candidates’ perceived knowledge of digital citizenship before and 
after the peer-teaching practice? 

4) What strategies and resources did teacher candidates plan to adopt from the workshops for teaching 
digital citizenship in their own classroom? 


Context of Study 

This study was conducted in the introductory course of an online instructional technology program in South 
Georgia. The course included a one-month-long project that required teacher candidates to work closely together in 
small groups of 4-5 members to design, develop, and deliver a 25-30 minute synchronous online professional 
development workshop related to digital citizenship using Blackboard Collaborate. The project goal was to increase 
their knowledge of digital citizenship through the workshop development. Each group was instructed to choose a 
theme of Digital Citizenship focusing on either “Digital Law” or “Digital Rights and Responsibilities” as the 
workshop topic. Teacher candidates could select any topic specific to their teaching context under these two themes 
(e.g., “Plagiarism, Intellectual Property, & Copyright - Cite It, Don’t Steal It!” or “Digital Footprint: What are you 
leaving behind?’’). Peers in the same class acted as learners for the workshops. To prepare for the workshop delivery, 
each group needed to complete the tasks including 1) small group discussions, 2) a project plan, 3) weekly progress 
reports, 4) an online session with the faculty to practice using Blackboard Collaborate, and 5) workshop materials 
and assessments. Faculty and peer feedback were provided throughout the group work process (e.g., suggestions for 
improvement for a project plan). Candidates also needed to complete a rubric evaluating their peers’ performance 
after attending the workshops. 


Research Design and Instruments 

A mixed method design was employed in the study, including survey investigation and examination of 
responses to the peer-evaluation rubric and workshop video archives. First, a 25-item anonymous survey was used to 
understand the candidates’ perceived effect of online synchronous peer-teaching practice on preparing them to teach 
digital citizenship in their own classroom. The survey structure was presented in Table 1. 


Table 1. Survey Structure 


Section | Content Questions | Question Type 
1 Gender Ql Demographic question 
2 Comfort level using Blackboard Q2-Q3 5-point Likert scale questions 


Collaborate, and comfort level with 
online synchronous peer-teaching 


3 Perceived effect of online Q4-Q7 5-point Likert scale questions 
synchronous peer-teaching practice 

4 Workshop instructor and learner roles | Q8-Q16 5-point Likert scale questions 

5 Self-rating of the knowledge of digital | Q17-Q20 | Q17-Q18: 4-point Likert scale 
citizenship questions; Q19: 5-point Likert scale 


question; Q20: Open-ended question 
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6 Best strategies and resources to teach 
digital citizenship 


Q21-Q25 


Open-ended questions 


Second, candidates used a rubric to rate their peers’ workshops based on the criteria including 1) 
organization & structure, 2) comprehensibility, 3) depth of thought, 4) design of workshop materials, and 5) 
preparation. They also had to explain what they learned from the each workshop. The responses to the rubric were 
examined using content analysis to understand what exact new concepts, strategies, or resources they have learned 
from the workshops. Third, the workshops were recorded and the archives were examined to understand what best 
strategies and resources had been utilized for teaching digital citizenship. 


Participants 

Twenty-eight teacher candidates who enrolled in the course were invited to participate in the study in the 
Spring 2019 semester. Candidates were assigned to six groups for workshop completion. Of 28 candidates, nine 
completed the survey voluntarily (including eight females and one male). For data analysis, each survey respondent 
was coded using an ID starting with S. Since the survey was anonymous, the researchers had no way to know who 
completed the survey. Therefore, the candidates who gave permissions for evaluation of rubric responses and 
workshop video archives were coded using different IDs starting with R. Groups were randomly named by color 
(e.g., Group Blue). Permission was received from nine candidates to examine their rubric responses (including eight 
females and one male) and five workshop video archives. The workshop topics and activities were presented in 
Table 2. 


Table 2. Workshop Topics and Activities 


Group | Topic Theme of Digital | Activities Included Blackboard Collaborate 
Citizenship Features and External 
Tools Used 
Blue How to maintain a Digital Rights and | Pre- and Post- Audio, Instant Polling, 
positive, Responsibilities Assessments, Mini Text Chat, Upload PPT, 
professional digital Lecture and and Whiteboard 
footprint as an Scenarios 
educator 
Green | Copyright Digital Law Pre- and Post- Audio, Text Chat, Web 
Assessments, Mini Tour, and Google Slides 
Lecture, and 
Scenarios 
Purple | Cyberbullying Digital Rights and | Pre- and Post- Audio, Emoticons, 
Responsibilities Assessments; Mini Instant Polling, Text 
Lecture, Videos, and | Chat, Web Tour, Upload 
Statistics PPT, and Google Form 
Yellow | Digital Footprint: Digital Rights and | Mini Lecture, Audio, Text Chat, Web 
What are you Responsibilities Videos, Reading Tour, Upload PPT, 
leaving behind? Activity, Hands-On Google Form 
Activity, Lesson 
Examples, and 
Workshop Survey 
Red Plagiarism, Digital Law Pre- and Post- Audio, Text Chat, 
Intellectual Assessments, Mini Google Slides 
Property, & Lecture, Videos, and | RQ Codes, and Google 
Copyright - Cite Workshop Survey Form 
It, Don’t Steal It! 
Results 


Among the nine survey respondents, two felt uncomfortable using Blackboard Collaborate and seven of 
them felt either comfortable or very comfortable using Blackboard Collaborate. S1 provided an explanation about 
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why she felt uncomfortable with Blackboard Collaborate: “I was nervous at first because I am not comfortable using 
Blackboard Collaborate so I was worried I would have technical issues, but it went much better that I expected.” In 
addition, two of the candidates felt uncomfortable with online synchronous peer-teaching, and at the same time, 
seven of them felt either comfortable or very comfortable with online synchronous peer-teaching. Thus, overall, the 
majority of the participants had a high comfort level with using Blackboard Collaborate or online synchronous peer- 
teaching. The data from the survey responses, peer evaluation rubric responses and workshop archives were 
integrated together and presented by each research question. 


Research Question One 

The research question one was to understand teacher candidates’ perceived effect of the online synchronous 
peer-teaching practice on preparing them to teach digital citizenship in their own classroom. There were four survey 
questions (Q4-Q7) related to this research question. Q4 related to learn-instructor interaction (M = 4.44, SD = .53), 
which examined if candidates learned to engage their learners in dialogues or provide learners timely feedback 
during the workshop. Q5 covered learner-learner interaction (M = 4.44, SD = .53). It examined if candidates were 
able to facilitate interaction and collaboration among learners during the workshop. Q6 asked about learner-content 
interaction (M = 4.44, SD = .53). It examined if candidates were able to engage learners with workshop content and 
activities. Q7 concerned about learner-interface interaction (M = 4.44, SD = .53). It examined if candidates were 
able to assist learners in using Blackboard Collaborate effectively for learning during the workshop. The means for 
these four questions were all above 4, which indicated that students agreed with the helpfulness of the online 
synchronous peer-teaching practice to support them to teach digital citizenship. 


Research Question Two 

The research question two was to explore what teacher candidates had learned in the peer-teaching practice 
from acting as both a workshop instructor and a learner. There were six survey questions (Q8-Q13) related to the 
instructor role in the workshop. According to the descriptive statistics, the means for these six questions were all 
above 4, which indicated that candidates agreed acting as a workshop instructor enhanced their ability to 1) plan for 
clear instruction of digital citizenship to learners (= 4.44, SD = .53), 2) handle unexpected instructional situations 
(M = 4.33, SD = .50), and 3) utilize technology effectively for teaching digital citizenship (VM = 4.44, SD = .53). 
Acting as a workshop instructor also 1) allowed the candidates to practice their strategies to teaching digital 
citizenship (M = 4.44, SD = .53), 2) improved their strategies to teach digital citizenship through learners’ feedback 
(M = 4.33, SD = .71), and 3) increased their self-confidence in teaching digital citizenship (M = 4.44, SD = .53). For 
example, S6 confirmed that she did learn “how to collaborate online and take the role of an instructor in online 
platforms” from the workshop. 

There were three survey questions (Q14-Q16) related to the learner role in the workshop. According to the 
descriptive statistics, the means for these three questions were all above 4, which indicated that candidates agreed 
acting as a workshop learner allowed them to learn new strategies to teach digital citizenship from their peers (M = 
4.44, SD = .53) and affirm their own strategies to teach digital citizenship (M = 4.33, SD = .71). In addition, acting 
as a workshop learner also gave them a chance to reflect on their own strategies to teach digital citizenship (M = 
4.44, SD =.53). 


Research Question Three 

The research question three was to explore if there were any changes to teacher candidates’ perceived 
knowledge of digital citizenship before and after the peer-teaching practice. There were four survey questions 
related to this research question. First, candidates were asked to rate their perceived knowledge of digital citizenship 
before and after the workshop using two 4-point Likert scale questions (Q17-Q18). According to descriptive 
statistics, the rating for the perceived knowledge of digital citizenship after the workshops (M = 3.56, SD = .53) was 
higher than the rating for the perceived knowledge of digital citizenship before the workshops (M = 2.33, SD = .50). 
Candidates perceived that they became either knowledgeable or very knowledgeable about digital citizenship after 
the workshops. In addition, according to the responses to Q19, candidates agreed that the workshops helped increase 
their knowledge of digital citizenship (M = 4.44, SD = .53). 

Six survey respondents for Q20 indicated that they learned new knowledge of digital citizenship. For 
example, S1 mentioned, “I was familiar with many of the topics covered, but I did learn a lot from the workshops. I 
learned in depth information on several topics including the definitions of many new vocabulary words that I was 
not aware of before.” S2 also expressed that she gained knowledge of digital citizenship from the workshops. She 
said, “I learned some great information and statistics regarding cyberbullying, plagiarism, copyright, and digital 
footprints. The workshops were very informative and eye-opening.” 
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Candidates provided more details about the new knowledge they learned in their rubric responses. The 
concepts mentioned repeatedly included copyright, cyberbullying, digital footprint and plagiarism. For example, R5 
said, “I did not know there were so many different types of cyberbullying. 47% of young people are cyberbullied. 1 
in 4 young people have been bullied more than once.” Not only did the respondents learn new knowledge, they also 
learned resources they could possibly apply to their own classroom. Just as R3 mentioned, “I learned of different 
lessons that can be used to bring awareness of the digital footprint for middle and high school students.” 


Research Question Four 

The research question four was to understand if there were any useful strategies and resources that teacher 
candidates could adopt to teach digital citizenship in their own classroom. Concerning the useful strategies and 
resources used to teach digital citizenship, real-life examples/scenarios, videos, statistics, assessments, and hands-on 
activities were repeatedly mentioned in both the survey responses and rubric responses (see Table 3). For example, 
Group Blue shared the scenarios of professional digital footprint in the workshop, which helped their peers 
understand how to maintain a positive digital footprint. Another example was that Group Purple shared a video of 
Amanda Todd (https://www.youtube.com/watch?v=ni- Y3wU92iU) to raise learners’ attention to the serious 
consequences of cyberbullying. 


Table 3. Useful Strategies Used to Teach Digital Citizenship 


Strategy Frequency of | Quote 

Occurrence 
Real-life 7 S2- “One effective strategy is to use real-life examples and make 
examples/scenarios connections.” 


R8- “This group shared some great tips on maintaining a positive 

digital footprint. They shared some great scenarios to show how 

leaving digital footprints can be tricky depending on the 

situation.” 

Videos 6 S2- “T believe links and videos are the most effective resources.” 

R3- “Providing videos and examples of how cyber bullying can 

have devastating effects in someone’s life and legal 

consequences, was an eye opening.” 

R7- “The ten types of cyberbullying were very helpful, and the 
video was very powerful.” 

Statistics 5 R2- “Use statistics to bring the topics to life.” 

R8- “I learned about the statistics related to cyberbullying. There 

was a much lower report rate than I initially thought. 48 states 

actually have defined laws against cyberbullying except for 

Wisconsin and Alaska. We have a 13% report rate in the state of 

Georgia. There is only a 16% overall. The highest percentage 

comes from harassing/threatening comments while the lowest 

percent comes from cruel pictures.” 

Assessments 5 S2- “Online quizzes are also great resources if they relate to the 

workshop topic. Polling and assessing throughout the workshop 

helps keep the audience engaged.” 

R3- “Pre and post assessments were done to engage the 

audience.” 

Hands-on activities | 3 R3- “It was interesting to learn how searches play such a big part 

of our daily lives.” 


Discussion 


There were several findings from the current study. First, students agreed with the helpfulness of the online 
synchronous peer-teaching practice to support them to teach digital citizenship. Second, the online synchronous 
peer-teaching practice allowed teacher candidates to play as both an instructor and a learner when learning digital 
citizenship. As acting as an instructor, they learned how to 1) plan for clear instruction of digital citizenship, 2) 
handle unexpected instructional situations during the practice, and 3) utilize technology effectively for teaching 
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digital citizenship. In addition, through the online synchronous peer-teaching practice, they practiced their strategies 
to teach digital citizenship and improved their strategies through learners’ feedback. The practice also helped 
increase their self-confidence in teaching digital citizenship. As acting as a learner, they learned new strategies to 
teach digital citizenship from their peers and affirm their own strategies. The practice also gave them a chance to 
reflect on their own strategies to teach digital citizenship. Third, candidates perceived that the practice helped 
increase their knowledge of digital citizenship. They agreed that they did learn new knowledge and resources they 
could possibly apply to their own classroom. Fourth, the use of real-life examples/scenarios, videos, statistics, 
assessments, and hands-on activities enhanced learning during the practice. Overall, the candidates’ learning 
experiences through online synchronous peer-teaching practice were positive. Just as S6 said, “I think this class 
supported my learning in encaging, learning and teaching digital citizenship.” 


Conclusions 


According to the results, the peer-teaching practice with synchronous technologies helped prepare teacher 
candidates to teach digital citizenship. They learned new knowledge, strategies, and resources from the practice and 
recognized its helpfulness to their learning about digital citizenship. The sample size of the current study was small, 
which made it difficult to generalize the findings to a large population. However, it could still provide insights into 
preparing teacher candidates to teach digital citizenship using an online synchronous peer-teaching practice. 
Directions for future research could include more samples, employing interview techniques to explore candidates’ 
perception in detail and using a pre- and post- assessment to measure the actual changes to candidates’ knowledge of 
digital citizenship. 
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Abstract 


In this study, the authors analyzed 51 articles published between 2015 and 2018 inclusive in the 
Journal of Online Learning Research (JOLR). The purpose of this study was to examine the trends regarding 
article topics, geography, research methods and article types, authorship, and citation frequency. The results 
indicated that JOLR gave additional attention to K-12 blended learning; compared to the field overall. Another 
common topic was professional development, with one special issue and the majority of top-cited articles 
related to this topic. Most of the studies were conducted in the United States, by researchers also located in the 
US. Finally, more than half of the studies employed inferential and interpretive methods. Future research is 
needed to examine if the trends from this study continue over a more extended period and if these results reflect 
the development of and change in the field of K-12 online and blended learning. 


Introduction 


In 2011, Dr. Richard West introduced a new series called “Journal Analysis Series,” which was 
published in the magazine Educational Technology. At the time, West (2011) wrote that to better understand 
the field of educational technology it was “helpful to review some of the journals that publish work in this area 
to see what conversations are being held, research being conducted, tools being developed, and theories being 
accepted” (p. 60). Later West (2016) reported that he and graduate students from his department had published 
23 articles in Educational Technology over a period of five years, “each analyzing a decade of scholarship in 
that journal” (p. 41), with each analysis providing a meta-discourse of insights. 

As a part of this series several distance/online learning journals were analyzed (e.g., American Journal 
of Distance Education; Distance Education; and Journal of Distance Education). However, as Barbour (2011) 
reported, only a small percentage of articles in these more general distance/online learning journals — such as 
those listed above — focused on the K-12 environment. In 2015 the Journal of Online Learning Research 
(JOLR) was established to publish articles related “to the theoretical, empirical, and pragmatic understanding of 
technologies and their impact on pedagogy and policy in primary and secondary (K-12) online and blended 
environments” (Association for the Advancement of Computing in Education, 2018, 1). Four years later, 
Armesen, Hveem, Short, West, and Barbour (2019) confirmed that JOLR was responsible for approximately 
7.0% of the 356 K-12 online learning journal articles they reviewed, and responsible for 41% of the articles 
published between 2015-17. 

Given the growing importance of JOLR as a publication outlet for scholars of and research into K-12 
online and blended learning, it is important to begin to understand the topics, types of articles, authors, and top- 
cited articles of this journal in an effort to provide insight into the larger context of the field. As such, the 
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purpose of this study was to conduct an analysis of JOLR with respect to article topics, research methods and 
article types, authors, and citations using procedures similar to those used in the “Journal Analysis Series.” 


Literature Review 


The practice of K-12 online learning traces its roots to around 1991 (Barbour, 2013; Clark, 2013). 
However, as Arnesen et al. (2019) reported, the first journal article focused on K-12 online learning was not 
published until 1996. Much of the literature and research that was initially published in the field came in the 
form of documents from private research centers and think tanks, individual program evaluation reports, and 
Master’s theses and doctoral dissertations (Barbour & Reeves, 2009; Cavanaugh, Barbour, & Clark, 2009). In 
fact, it wasn’t until 2006 that the number of journal articles related to K-12 online learning began to grow in 
larger numbers (Arnesen et al., 2019). 

To date, there have been four comprehensive literature reviews that have been published in the field of 
K-12 online learning (Barbour & Reeves, 2009; Cavanaugh et al., 2009; Hasler Waters, Barbour, & Menchaca, 
2014). Overall, the themes in each of these articles have been fairly consistent. For example, the dominant 
theme in each of the four literature reviews has been that the majority of research into K-12 online learning has 
been focused on comparing how students performed based on the delivery model of their learning (i.e., 
distance/online vs. brick-and-mortar). The remaining literature has tended to focus on: 1) components of 
teaching and learning online (Barbour & Reeves, 2009; Cavanaugh et al., 2009) or 2) online learning policy, 
particularly as it relates to full-time K-12 online learning (Barbour & Reeves, 2009; Hasler Waters et al., 2014). 

In addition to these thematic literature reviews, there have also been several different analyses of the 
state of the field of K-12 online learning. For example, Barbour (2007) examined the backgrounds of various 
researchers who had published in the field and found that many of these early scholars in K-12 distance/online 
learning came from varied backgrounds, had very different professional training, and were working in a wide 
variety of disciplines. Lowes (2014) examined the research methods used in select journal articles from 2004 to 
2014 in online teaching and learning. She reported that initial research in the field utilized experimental or 
quasi-experimental methods, and then transitioned to small scale case studies and survey-style research. 

In their contribution to the second edition of the Handbook on K-12 Online and Blended Learning, 
Lokey-Vega, Jorrin-Abellan, and Pourreau (2018) studied the ‘Research Clearinghouse for K-12 Blended and 
Online Learning’ to determine if any learning theories were included in the title, abstract, or keywords of the 
research contributed to that database. Lokey-Vega and her colleagues concluded that while “the relatively young 
field of research in K-12 online learning has achieved much in establishing ourselves as a distinct and definable 
group” (p. 85), the body of research had just started to “stretch [the] field to seek and understand instances of 
success and test well-supported historically-important distance learning theories” (p. 85). Later in the same 
handbook, Lowes and Lin (2018) found that “much of the early research on online learning at the K-12 level 
focused on comparing online supplemental courses with their face-to-face counterparts... [while] the rest of the 
research falls under the broad heading of studies of particular cases” (p. 92). 

As acomplete volume, the second edition of the Handbook on K-12 Online and Blended Learning 
would suggest to readers that the field was largely atheoretical, focused on comparing student performance in 
online and brick-and-mortar environments or conducting case studies on a variety of aspects in K-12 online and 
blended learning, and primarily concerned with the United States. This is not inconsistent with the findings of 
Barbour (2018), who reported that to date researchers in the field of K-12 distance, online, and blended learning 
rarely used theoretical or conceptual frameworks to guide their research, did not use validated instruments as a 
part of the research tools, and — in many instances — failed to define the characteristics of what was being 
researched. This final point is important, as the nature of the distance, online, or blended learning contexts can 
vary significantly from one setting to another. Without adequately describing the characteristics of that setting, it 
renders meaningful comparisons with future research difficult to impossible. 

It should be noted that all of the literature discussed thus far has focused on K-12 distance/online 
learning. However, the field is generally viewed in broader terms as the field of K-12 distance, online, and 
blended learning (Kennedy & Ferdig, 2018). Reasons for the lack of coverage of K-12 blended learning include 
the general lack of literature and, in particular, research on the topic. 

There have even been some that have argued that the focus on K-12 blended learning by those in the 
larger field of K-12 distance, online, and blended learning is based on ideological or political motivations 
(Barbour, 2014); arguing that outside of the United States blended learning is seen as a form of technology 
integration at the K-12 level. However, the Association for the Advancement of Computing in Education, the 
organization that publishes JOLR, is based in the United States. Additionally, according to the founding editors, 
the purpose of JOLR was to “address online learning... [and] educators who have chosen to blend online 
learning tools and strategies in their face-to-face classroom” (Kennedy & Archambault, 2015, p. 6). As such, our 
analysis of JOLR will focus on the complete field of distance, online, and blended learning. 
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Methodology 


To complete our analysis, we reviewed and analyzed all 51 articles published in JOLR between 2015 
and 2018, excluding editorials and book reviews, for trends in topics, article types, authorship, geography, and 
citations, using procedures consistent with the Journal Analysis Series (West, 2011, 2016). 


Article Topics and Geography Analysis 

The website for JOLR (see https://www.aace.org/pubs/jolr/ ) provided abstracts for all of the articles 
that we analyzed. We entered the 51 abstracts into Textalyser (see http://textalyser.net/), an online text analysis 
tool, which extracted the top relevant two- and three-word phrases to determine the journal’s central topics. 
Additionally, we reviewed the abstract and methodology portions of each article to determine whether they had 
a specific focus on school level, such as high school and elementary school, or had no focus. It should be noted 
that throughout JOLR’s history there have been several special issues. 


Table 1. JOLR Special Issues 


Year Volume Issue Special Issue Topic 

2015 1 1 * 

2016 2 2 Professional development 
2016 2 4 Supporting students 

2017 3 1 Bended learning 

2018 4 1 Students with disabilities 
2018 4 3 Online course design 


* While not a special issue, the inaugural issue featured invited works from several of the scholars who helped 
to create the journal (Kennedy & Archambault, 2015). 

We mention these special issues, as well as the invited inaugural issue, to allow readers to better 
understand some of the topical trends, as the inclusion of a special issue on a particular topic has the ability to 
skew the number of articles focused on that topic in the overall data. 

We also examined each of the articles to determine either the geographic location of the study or the 
geographic focus of the article. Generally, the geographic focus was referenced in the abstract and/or 
methodology portions of the article, although the entire article was reviewed to determine the geographic focus. 


Article Types and Methodologies Analysis 

We used a similar coding methodology that West and his colleagues used in the Journal Analysis Series 
(see West, 2011, 2016). Each article was coded according to six possible categories to identify the 
methodological trends. We used the following coding definitions: 

e Descriptive: The research primarily relies on data collected from surveys and reports where the 
statistics are descriptive in nature, such as means, averages, and percentages. 

e Inferential: The research includes studies that are quasi-experimental, experimental, or correlational, or 
that validated a survey instrument using factor analysis or item response. They report inferential 
statistics that test hypotheses or report differences between groups. 

e Interpretative: These studies include case studies, ethnographies, interview studies, observation studies, 
and document analysis studies. These articles often focus on interpreting data to develop theory and are 
more substantial than categorizing simple answers to a one- or two-question open-ended survey. 

e Theoretical: The research is not data-based, but includes discussions of new theories, models, 
instructional approaches, designs, and reviews of literature. 

e Content analysis: The research presents discrete categories and labels data according to deductively 
categorize data. 

e Combined methods: The research uses any two combinations of methods, most commonly interpretive 
and inferential methods. 

We assigned each article to a single category, and those articles with elements of more than one category were 
assigned according to their primary characteristics. 


Authorship Analysis 

We conducted the analysis of authorship by extracting the frequency of each author’s contributions. 
We then employed a point scale, giving first authors three points; second authors, two points; and third authors 
and beyond, one point. 

We also examined each of the articles to determine the geographic location of the author based on their 
institutional location. The JOLR website provides the authors’ names, each author’s institutional affiliation, and 
the country where each institutional affiliation is located. We reviewed this information and categorized the 
authors according to the country of their institutional affiliations. 
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Citation Analysis 

We analyzed Google Scholar, using the Publish or Perish software (Harzing, 2010), to examine the 
number of citations for each article as of 31 December 2018. We analyzed all 51 articles published between 
2015 and 2018 to identify the 11 top-cited articles, each of which was cited seven or more times. 


Results 


This study analyzed 51 articles published in JOLR from 2015 to 2018. The findings below will identify 
trends regarding article topics, article types and research methods, authors, and citations, using procedures like 
those used in the Journal Analysis Series (see West, 2011, 2016). In addition, this study will add two new 
domains: the article geography analysis and author geography analysis. 


Article Topics and Geography 

Table 2 indicates the most frequent three- and two-word phrases appearing in the article abstracts. 
Those phrases referenced fewer than three times for three-word phrases and fewer than 15 times for two-word 
phrases are not listed in this table. 


Table 2. Frequency of phrases in abstracts 


3-word phrases Number of articles Phrase count 
Face-to-face 8 9 
Online and blended 5 7 
Online course design 5 6 
Blended learning environment(s) 4 6 
Online learning environment(s) 4 =) 
Students with disabilities 3 6 
Community of engagement 2 4 

2-word phrases Number of articles Phrase Count 
Blended learning 14 29! 
Online teacher(s) 13 29 
Online courses(s) 12 257 
Professional development 12 23 
High school 9 19 
Online Learning 8 93 
Learning environment(s) 5 8 


“Blended learning” and “online teacher(s)” were the most frequent topic phrases. Both had 29 instances within 
the abstracts, with 14 and 13 articles, respectively. The phrase “blended learning” was also found in “blended 
learning environment(s),” which added another six instances in four articles. Additionally, in many instances the 
phrase “blended learning” was preceded by “online and”. 

The other topics — “face to face,” “online and blended,” “students with disabilities,” “communities of 
engagement,” “professional development,” and “high school” were all included in articles that focused on some 
aspect of online and/or blended learning. The four articles that did not contain direct references to online or 
blended learning in the abstracts did, nevertheless, discuss some aspect of online or blended learning. 

Table 3 shows the specific focus on school level of each article. The number of articles that have no 
school level focus is also listed at the bottom of the table. 

Table 3. Specific focus on school level 


Specific levels Number of articles 
High school/secondary 23 
K-12 17 
Middle School 7 
Elementary School 2 
K-8 1 
No focus on level 1 


1 This number excludes incidences of the phrase used in “blended learning environment(s).” 
2 This number excludes incidences of the phrase used in “online course design.” 


3 This number excludes incidences of the phrase used in “online learning environments.” 
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Fifty of the 51 articles in this study discussed different aspects or levels of K-12 contexts. Twenty-three articles 
focused on high schools or secondary schools, 17 articles focused on K-12 generally, seven articles focused on 
middle schools, two articles focused on elementary schools, and one article focused on K-8. The one article that 
had no specific school level focus (Evmenova, 2018) focused on professional. 

Table 4 reports the geographic location of the study or the geographic focus of the article itself. The 
number of articles that have no focus on geography is also listed at the bottom of the table. 


Table 4. Location where the study took place or where the article was focused. 


Geography No. of Studies 
United States 42 

Turkey i 

Brazil 1 

India 1 

Multiple countries 1 

No focus/Unable to determine 5 


There were five articles that had no specific geographic focus or we were unable to determine the location. With 
the exception of these five articles, the vast majority (i.e., 91%) of articles with a geographic focus published by 
JOLR focused on the United States. There was one article focused on each of the following: Turkey, Brazil, and 
India. All but one of the studies were conducted in the country of the author's institution. However, it is worth 
mentioning that the one study that did not follow this trend involved authors from multiple countries. 


Article Types and Methodologies 
Table 5 lists the different research method categories and the number of articles for each category. 


Additionally, this table describes the overall proportion of each methodology. 


Table 5. Categorization of all articles published 


Method Total Number Total Percentage 
Interpretive 16 31% 
Inferential 15 29% 
Combined 11 22% 
Theoretical 6 12% 
Content Analysis 2 4% 
Descriptive 1 2% 


The most common methodological category was interpretive. However, there was only one fewer inferential 
studies than interpretive studies. Content analysis and descriptive methods made up only a small proportion. 

Table 6 indicates both the number of articles per methodological category for each year and the yearly 
average for each category. To ensure precision, the yearly average was rounded to two decimal places. 


Table 6. Number of articles published each year according to category 


Method 2015 2016 2017 2018 Yearly Average 
Interpretive 4 7 2 3 4.99 
Inferential 3 6 3 3 3.75 
Combined 5 3 2 1 2.75 
Theoretical 1 1 0 4 1.50 
Content Analysis 0 0 1 1 0.50 
Descriptive 0 0 1 0 0.25 


Similar to Table 5, Table 6 shows that an interpretive approach was the most frequently used methodology, in 
terms of the yearly average. It is worth noting that the number of theoretical articles increased abruptly in 2018, 
increasing to four theoretical articles in 2018. Two of these four theoretical articles focused on the special issue 
topic of online course design, indicating that this topic may have been more aligned to this category. 
Additionally, the number of articles that applied combined methods tended to decrease gradually year by year. 


Authorship 
Table 7 lists all the authors who published two articles or more, ranked according to the medal system. 


Authors with three points or fewer were only listed according to the number of articles each published. 


Table 7. Overall authorship by number of articles and point 


Author name No. of articles No. of points 
Jered Borup 4 9 
Mark Stevens 3 7 


123 


Charles Graham 

David Adelstein 

Leanna Archambault 

Annisa Loky-Vega 

Michael Barbour 

Mary Rice 

43 authors 

33 authors 

39 authors 
The 51 articles had a total of 123 authors. Jered Borup was a top author in this journal with the most number of 
publications and the highest author rank. Jered Borup, David Adelstein, and Leanna Archambault each 
published two articles as the first author. Of these listed authors, it should be noted that Jered Borup and Leanna 
Archambault are current editors of JOLR. Interestingly, although 86% of the articles had two or more authors, 
most of the authors, approximately 93% of the authors published only one article. 

Table 8 shows the countries where the authors’ institutional affiliation were located. It also counts the 
total number of authors from each country. 
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Table 8. Geographical location of authors 


Geography No. of authors 
United States 115 
United Kingdom 5 
Turkey 3 

Brazil 2 
Canada 1 

India 1 


Similar to the results shown in Table 4 that the majority of the studies occurred in or were focused on the United 
States, 93% of authors’ institutional affiliations were located in the United States, which indicated that the 
geographic focus of the study was almost always aligned with the geographic location of the authors’ 
institutional affiliations. One of the exceptions to this pattern was when two authors from Brazil, together with 
one author from Canada, conducted a study set in Brazil. 


Citations 
Table 9 reports the top cited articles. Only articles with more than five citations are listed in this table. 


Table 9. Frequency of article citation 


Title 

Authors Type Citations Year 

Learning to learn online: Using locus of control to help students become successful online learners 

Susan Lowes, Peiyi Lin Inferential 22 2015 

Documenting and sharing the work of successful on-site mentors 

Joseph Freidhoff, Jered Borup, Rebecca Stimson, Kristen _ Interpretive 18 2015 

DeBruler 

Pioneering the digital age of instruction: Learning from and about K-12 online teachers 

Leanna Archambault, Jean Larson Combined 13 2015 
Methods 

Incremental progress: Re-examining field experiences in K-12 online learning contexts in the United States 

Leanna Archambault, Kathryn Kennedy, Catharyn Combined 11 2016 


Shelton, Medha Dalal, Laura McAllister, Sabrina Huyett Methods 

Building better courses: Examining the construct validity of the iNACOL national standards for quality 
online courses 

David Adelstein, Michael Barbour Theoretical 10 2016 
A call to action for research in digital learning: Learning without limits of time, place, path, pace...or 
evidence 


Cathy Cavanaugh, Christopher Sessums, Wendy Drexler _ Theoretical 8 2015 
Credit recovery in a virtual school: Affordances of online learning for the at-risk student 
Kevin Oliver, Shaun Kellogg Inferential 8 2015 


The status of middle and high school instruction: Examining professional development, social desirability, 
and teacher readiness for blended pedagogy in the southeastern United States 
Rebecca A. Parks, Wendy Oliver, Elaine Carson Inferential 8 2016 
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Listening to the teachers: Using weekly online teacher logs for ROPD to identify teachers’ persistent 
challenges when implementing a blended learning curriculum 


Jeremy Riel, Kimberly A. Lawless, Scott W. Brown Interpretive 8 2016 
Adapting the curriculum: How K-12 teachers perceive the role of open educational resources 
Beatriz de los Arcos, Robert Farrow, Rebecca Pitt, Martin Inferential 7 2016 


Weller, Patrick McAndrew 
An analysis of the curriculum requirements for K-12 online teaching endorsements in the U.S. 


Laura McAllister, Charles Graham Content Analysis 7 2016 
Fostering student success and engagement in a K-12 online school 
Heidi Curtis, Loredana Werth Interpretive 7 2015 


The citations listed in the table are reflective of those provided by Google Scholar as of December 31, 2018. As 
would be expected, the articles from 2017 and 2018 were too recent to have any meaningful citation counts, 
which was likely why there were no articles from those years in the table. Similarly, Jered Borup, who had 
published the most articles and received the highest author rank in Table 7, was also one of the most cited 
authors. In addition, David Adelstein, Leanna Archambault, Michael Barbour, and Charles Graham were among 
the most cited list. These top cited articles covered all methodological categories except the descriptive 
category. Finally, with the exception of Beatriz de los Arcos and her co-authors, whose institutional affiliation 
was in the United Kingdom, all other authors’ geographic locations were the United States. 


Discussion 


The results of the topic analysis indicated that both online and blended learning were important topics 
in JOLR, with blended learning heading the list. The interest in blended learning is inconsistent with previous 
literature that primarily focused on K-12 distance/online learning (Molnar et al., 2017). The efforts JOLR has 
made to bring blended learning prominently into the discussion of distant and online learning reflects their 
stated purpose to publish articles about both online and blended contexts. 

After blended learning, online teacher(s) was the other most frequent topic phrase. This phrase 
primarily appeared in the context of teacher preparation or professional development, which might indicate the 
critical role of the teacher, as well as the need for teacher preparation programs and professional development 
within K-12 online and blended learning environments. This result was consistent with Arnesen et al. (2019), 
which found teacher education was the most frequent two-word phrase and teacher education programs the most 
frequent three-word phrase. The results also revealed trends in course design for K-12 online and blended 
learning, as well as professional development for teachers in those same contexts. 

As might be expected from a journal that focuses on K-12 online and blended learning, all but one of 
the 51 articles in this study discussed different aspects or levels of K-12 contexts. Of these articles 23 focused 
specifically on high schools or secondary schools, seven on middle schools, two on elementary schools, and one 
on K-8. In their review of the literature on virtual schools, Barbour and Reeves (2009) revealed a similar pattern. 
They cited specifics of large growth in virtual high schools and in high school students’ participation in online 
learning, but cited elementary schools only in the larger context of K-12 learning. Similarly, 17 of the 51 articles 
do not focus on specific grade levels, but rather on the broader context of K-12 learning, suggesting that 
elementary and middle school contexts are still an important part of the overall concerns and interests of K-12 
learning. It is also interesting to note that 12 (or 24%) of the articles focus on some aspect of online and/or 
blended teachers’ training and experience. 

The article geographic analysis showed that 82% (i.e., 42 of 51) studies occurred or were conducted in 
the United States, which was consistent with the literature in general. As might be expected, the results also 
indicated that the geographic focus of the study was almost always aligned with the geographic location of the 
authors’ institutional affiliations. These results suggest that JOLR might benefit from a more global focus to 
achieve its commitment to being an international journal, and we note that the journal has recently added an 
“international section” to encourage these kinds of submissions (see http://www.aace.org/pubs/jolr/). 

We found interpretive articles accounted for 31% of all methodological categories, followed by 
inferential articles (i.e., 29%). The results generally aligned with Lowes’ (2014) study, in which she reported 
that the research in the field had transitioned from experimental or quasi-experimental methods to small-scale 
case studies and survey-style research. Further, according to Arnesen et al. (2019), theoretical methods were the 
most common type of K-12 online learning articles through 1994 to 2016, but the authors also reported that the 
number of interpretive and inferential articles was increasing. In fact, interpretive articles were found to be more 
common than theoretical articles between 2012 and 2016. Similarly, Barbour (2018) underscored the importance 
of interpretive and inferential methods, and suggested scholars should continue to focus on interpretive and 
inferential research in K-12 online and/or blended settings. However, Lokey-Vega, Jorrin-Abellan, and Pourreau 
(2018) claimed that theory played a pivotal role in the field of K-12 online and blended learning and advocated 
for an increase in the use of theory in the field. 
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Most of the top authors identified in this study, such as Michael Barbour, Charles Graham, Jered 
Borup, and Leanna Archambault, were also in the list of the top 20 authors in the recent study of Arnesen et al. 
(2019). This result supports the assessment by Armmesen and her colleagues that the field of K-12 online 
education is a ‘small research community,’ or at least those authors who were the most active represent a small 
field. Additionally, in accordance with Barbour’s (2007) study, these results also displayed a shift in that most 
of the top authors were from post-secondary institutions, as opposed to non-profit research organizations. The 
results also suggested that more new scholars were beginning to join the field, given the fact that 113 of the 
authors contributed only a single article. The key for the future growth of the field will be to ensure that these 
authors continue to contribute to the field. 

The top-cited articles focused on K-12 online and blended education, as did 92% of the entire body of 
articles as seen in the abstract analysis, both of which reflected the general aim and scope of JOLR. Specifically, 
the fact that six of 11 top-cited articles (i.e., 55%) were related to teacher education programs or teachers’ 
professional development in online and/or blended settings indicated a field trend in professional development 
for online and blended teachers. This finding was also consistent with a trend in the broader distance education 
literature that recent studies put an increasing emphasis on the issue of professional development (Lee, Driscoll, 
& Nelson, 2007). Similar to the findings of Arnesen et al. (2019) that many top authors were also among those 
whose articles were cited more frequently, the top authors including Jered Borup, Charles Graham, David 
Adelstein, Leanna Archambault, and Michael Barbour were also the most cited authors. This result further 
supported the comment by Arnesen and her colleagues that the studies conducted by the most productive 
authors were also the most significant — at least from a citation perspective. Additionally, more than half of the 
most cited articles used inferential or interpretive methods, while only two of 12 articles were theoretical 
articles. This result was consistent with what was found in some of the articles produced from the Journal 
Analysis Series (West 2011, 2016). Interestingly, in his summary of the formal project, West (2016) remarked 
that “some journals seem to have discouraged or even disallowed theoretical work, including literature 
syntheses, in an effort to focus only on empirical work. However, we consistently found theoretical/literature- 
based articles to be among the most cited in a journal” (p. 44). This inconsistency might indicate that a shift has 
emerged from the early preference in theoretical methods — consistent with other educational technology 
disciplines — to data-based methods, especially inferential and interpretive, within the field of K-12 online and 
blended learning. 

It should be noted that there are several limitations in this study. First, we excluded the keyword 
analysis because only 16 out of 51 articles (i.e., 31%) had keywords, which was not a large enough sample to 
generalize the primary topics for the whole journal. Second, since JOLR is a very young journal, there were only 
51 articles over the four- year period that could be analyzed. In contrast, in the Journal Analysis Series, West 
(2011) indicated that the goal for the series was to examine a decade of scholarship from each of the selected 
journals. However, it is important to note that Lokey- Vega (2018) felt that the establishment of JOLR was one 
of several steps that “scholarly leaders [had undertaken to] establish critical community infrastructure for 
networking and knowledge building that would benefit any current or new scholar in the field” (p. 4). 


Conclusions and Implications 


In general, JOLR complies with its aims to publish articles related “to the theoretical, empirical, and 
pragmatic understanding of technologies and their impact on pedagogy and policy in primary and secondary (K- 
12) online and blended environments” (Association for the Advancement of Computing in Education, 2018, 4 
1). Specifically, according to our analysis, the additional attention JOLR has given to blended contexts sets it 
apart from previous literature that focused primarily only on K-12 online learning. JOLR also raises awareness 
of the importance of teachers’ professional development in online and blended environments. Second, we found 
that most of the studies on K-12 online and blended learning occurred or were conducted in the United States. 
Similarly, most of the researchers’ institutional affiliations were in the United States. Third, the fact that 115 of 
123 authors in our study had published only one article may indicate that the field is attracting new scholars or 
that scholars who have other research agendas are finding that their interests overlap with issues in the K-12 
context. Finally, JOLR favored inferential and interpretive articles. 

As the only journal that primarily publishes articles specializing in K-12 online and/or blended 
learning, JOLR provides a unique platform for researchers, especially new researchers to this field, to present 
their studies on K-12 online and/or blended learning in a journal focused on the subject rather than present them 
scattered among a variety of journals. However, given that the field is still a relatively new research area, future 
research can continue to examine the trend analysis over a more extended period to reflect the development of 
and change in the field of K-12 online and blended education. Further, researchers outside of the United States 
should be encouraged to conduct research that focuses on other countries in addition to the United States, thus 
facilitating comparisons between the implementation and use of online and blended approaches in different 
countries and cultures, as well as encouraging diversification of the field. 


126 


References 


Archambault, L., & Borup, J. (2017). Delving further into blended learning: Examining professional 
development models, a flipped learning approach, and open educational resources. Journal of Online 
Learning Research, 3(2), 103-104. Retrieved from https://www.learntechlib.org/primary/p/178543/ 

Arnesen, K. T., Hveem, J., Short, C. R., West, R., & Barbour, M. K. (2019). K-12 online learning journal 
articles: Trends from two decades of scholarship. Distance Education, 40(1), 32-53. 
https://doi.org/10.1080/01587919.2018.1553566 

Association for the Advancement of Computing in Education. (2018). Journal of Online Learning Research: 
About JOLR. Waynesville, NC: Author. Retrieved from https://www.aace.org/pubs/jolr/ 

Barbour, M. K. (2007). Who’s researching virtual schools: A case for instructional technologists. Distance 
Learning, 4(1), 7-12. 

Barbour, M. K. (2013). The landscape of K-12 online learning: Examining what is known. In M. G. Moore 
(Ed.), Handbook of distance education (3" ed.) (pp. 574-593). New York: Routledge. 

Barbour, M. K. (2018). Lessons for K-12 distance, online and blended learning from research in higher 
education. Lansing, MI: Michigan Virtual Learning Research Institute at Michigan Virtual University. 
Retrieved from https://mvlri.org/research/publications/examining-online-research-in-higher-education- 
what-can-we-replicate-in-k-12/ 

Barbour, M. K., & Reeves, T. C. (2009). The reality of virtual schools: A review of the literature. Computers 
and Education, 52(2), 402-416. https://doi.org/10.1016/j.compedu.2008.09.009 

Cavanaugh, C., Barbour, M. K., & Clark, T. (2009). Research and practice in K-12 online learning: A review of 
open access literature. International Review of Research in Open and Distance Learning, 10(1). 
Retrieved from http://www.irrodl.org/index.php/irrod|/article/view/607 

Clark, T. (2013). The evolution of K-12 distance education and virtual schools. In M. G. Moore (Eds.), 
Handbook of distance education (3" ed.) (pp. 555-573). New York: Routledge. 

Evmenova, A. (2018). Preparing teachers to use universal design for learning to support diverse learners. 
Journal of Online Learning Research, 4(2), 147-171. Retrieved from 
https://www.learntechlib.org/primary/p/181969/ 

Hasler Waters, L., Barbour, M. K., & Menchaca, M. P. (2014). The nature of online charter schools: Evolution 
and emerging concerns. Journal of Educational Technology & Society, 17(4), 379-389. Retrieved from 
https://www.j-ets.net/ETS/journals/17_4/26.pdf 

Harzing, A. W. (2010). The publish or perish book: Your guide to effective and responsible citation analysis. 
Melbourne, Australia: Tarma Software Research Pty Ltd. Retrieved from 
https://harzing.com/publications/publish-or-perish-book 

Kennedy, K., & Archambault, L. (2015). Bridging the gap between research and practice in online learning. 
Journal of Online Learning Research, 1(1), 5-7. Retrieved from 
https://www.learntechlib.org/p/149914/ 

Kennedy, K., & Ferdig, R. (2018), Handbook of research on K-12 online and blended learning (2™ ed.). 
Pittsburgh, PA: Entertainment Technology Center Press, Carnegie Mellon University. Retrieved from 
https://figshare.com/articles/Handbook_ of Research_on_ K- 

12 Online and Blended Learning Second Edition /6686813 

Lee, Y., Driscoll, M. P., & Nelson, D. W. (2007). Trends in research: A content analysis of major journals. In 
M. G. Graham (Ed.), Handbook of distance education (2" ed.) (pp. 31-41). Mahwah, NJ: Lawrence 
Erlbaum. 

Lokey-Vega, A. (2018). Background and historical perspectives. In K. Kennedy & R. Ferdig (Eds.), Handbook 
of research on K-12 online and blended learning (2" ed.) (pp. 3-6). Pittsburgh, PA: Entertainment 
Technology Center Press, Carnegie Mellon University. Retrieved from 
https://figshare.com/articles/Handbook of Research_on K- 

12 Online and Blended Learning Second Edition /6686813 

Lokey-Vega, A., Jorrin-Abellan, I. M., & Pourreau, L. (2018). Theoretical perspectives in K-12 online learning. 
In K. Kennedy & R. Ferdig (Eds.), Handbook of research on K-12 online and blended learning (2 
ed.) (pp. 65-90). Pittsburgh, PA: Entertainment Technology Center Press, Carnegie Mellon University. 
Retrieved from https://figshare.com/articles/Handbook of Research_on K- 

12 Online and Blended Learning Second Edition /6686813 

Lowes, S. (2014). A brief look at the methodologies used in the research on online teaching and learning. In R. 
Ferdig & K. Kennedy (Eds)., Handbook of Research on K-12 Online and Blended Learning (pp. 83- 
104). Pittsburgh, PA: Entertainment Technology Center Press, Carnegie Mellon University. Retrieved 
from https://figshare.com/articles/Handbook_ of Research on K- 

12 Online and Blended _Learning/6686810 


127 


Lowes, S., & Lin, P. (2018). A brief look at the methodologies used in the research on online teaching and 
learning. In K. Kennedy & R. Ferdig (Eds.), Handbook of research on K-12 online and blended 
learning (2™ ed.) (pp. 91-110). Pittsburgh, PA: Entertainment Technology Center Press, Carnegie 
Mellon University. Retrieved from https://figshare.com/articles/Handbook of Research on K- 
12 Online and Blended Learning Second Edition /6686813 

Mohammed, S. (2017). Special issue: Connecting research and practice to understand efficacy in K-12 blended 
learning. Journal of Online Learning Research, 3(1), 1-4. Retrieved from 
https://www.learntechlib.org/primary/p/177971/ 

West, R. E. (2011). About this article and new series. Educational Technology, 51(4), 60. 

West, R. E. (2016). Insights from the journal analysis series: What we have learned about educational 
technology research. Educational Technology, 56(1), 41-45. 


128 


A Comparison of Preservice and Inservice Teachers’ Most-Valued 
Technology-Supported Activities for Teaching and Learning 


Joan E. Hughes 
The University of Texas at Austin 
Learning Technologies Program 
Department of Curriculum and Instruction 
1912 Speedway, Mailstop D5700, Sanchez 244M 
Austin, Texas 78712 


Yin Hong Cheah 
The University of Texas at Austin 
Learning Technologies Program 
Department of Curriculum and Instruction 
2501 Lake Austin Blvd. Austin, Texas 78703 


Yi Shi 
The University of Texas at Austin 
Learning Technologies Program 
Department of Curriculum and Instruction 
1620 W 6" St., Austin, Texas 78703 


Kuei-Hui Hsiao 
The University of Texas at Austin 
Learning Technologies Program 
Department of Curriculum and Instruction 
1912 Speedway, Stop D5700, Sanchez 244M 
Austin, Texas 78712 


Keywords: technology integration, pedagogical reasoning 
Abstract 


We examined teachers’ pedagogical reasoning for and the technological knowledge underlying their most- 
valued technology-supported activities for teaching and learning. Data from 140 preservice and 100 inservice 
teachers included open-ended, narrative responses to survey questions. Qualitative research methods guided analysis 
of the data that identified (a) the technology-supported activities and (b) the technical tools, target users, types of 
uses, rationales for use, and the technological pedagogical content knowledge (TPACK) underlying each activity. 
Preservice teachers described mostly teacher-focused and fewer student-focused techno-activities, and their 
reasoning for use focused on the technology’s presentational and engagement effects. A majority of inservice 
teachers’ techno-activities were student-focused, and their reasoning highlighted the technology’s support for 
knowledge acquisition of higher-order cognitive skills and collaborative learning. The knowledge underlying all 
teachers’ techno-activities was predominantly technological pedagogical knowledge (TPK), but inservice teachers 
also evidenced technological content knowledge (TCK). These results may reveal differences in the teachers’ 
respective learning experiences or reflect a professional maturation process. 


Introduction 


Teachers are the decision-makers and the designers of whether and how they use technology in their 
classrooms (Ertmer, 2005; Tsai & Chai, 2012). Such decision-making and designing occurs within complex 
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educational contexts. Teachers face a range of barriers and supports for technology use in classrooms (e.g., Authors, 
2016; Ertmer, 1999, 2005; Ertmer & Ottenbreit-Leftwich, 2010; Hew & Brush, 2007; Tsai & Chai, 2012). Increased 
clarity on barriers has guided shifts in teacher education and professional learning that have led to increased 
technology use by teachers over the last decade. Yet, research continues to show preservice and inservice teachers 
may not have a wide breadth of technology usage in subject areas, and technology integration has yet to be prevalent 
and equitable in PK-12 schools (e.g., Tondeur, Pareja Roblin, van Braak, Voogt, & Prestridge, 2017; Zyad, 2016). 

Decades of research has focused on describing teachers’ technology knowledge, often using the 
technological pedagogical content knowledge (TPACK) conceptual model (Angeli & Valanides, 2005; Mishra & 
Koehler, 2006). Recent studies find that teachers’ percetved TPACK may not aptly reflect their behavioral intentions 
or smoothly translate into enacted actions in the classroom (e.g., Jen, Yeh, Hsu, Wu, & Chen, 2016). Thus, research 
must begin to unpack teachers’ technology-mediated practices to better understand teachers’ TPACK development 
and use in the PK-12 environment, such as by considering the role of pedagogical reasoning in the context of 
technology (e.g., Harris & Phillips, 2018; Hofer & Harris, 2019). There is a range of teachers’ rationalizations 
underlying their choices of technology (Li, 2014; Voet & Wever, 2017). A teacher’s reasoning for technology use 
may reveal how they value technology applied in their practice. Yet, the research that links teachers’ technology use 
or planned use with their underlying reasoning for their technology choice(s) is sparse. Hence, our study examined 
teachers’ reasoning for using technologies that they deemed valuable for teaching and learning. 


Literature Review 


Technology Use by Inservice and Preservice Teachers 

Inservice teachers in PK-12 contexts use technology mainly for lesson preparation and professional 
development needs (Ottenbreit-Leftwich, Glazewski, Newby, & Ertmer, 2010; Zyad, 2016). For instance, Zyad’s 
(2016) study showed that technology was most frequently used by teachers for preparing lesson plans and keeping a 
database of various instructional materials. In recent years, more research has documented the increasing use of 
technology by inservice teachers during their classroom instructional time. The majority of these classroom-based 
technology uses were teacher-focused and transmissive in nature (Tondeur et al., 2016; Voet & De Wever, 2017). 
For example, Tondeur et al. found technology was integrated by most of the teacher participants to support 
structured learning approaches—most commonly the use of the data projector or interactive whiteboard to deliver 
instruction. Few opportunities were provided for student-centered applications. Similar findings were observed by 
Voet and De Wever. Their results showed that student use of technology remained scarce as teachers perceived 
technology as a resource for the learning task, rather than as a tool for supporting student inquiry learning. 

In comparison to inservice teachers, there are fewer studies of preservice teachers’ technology use. Among 
the studies of technology use, similar patterns, such as the domination of productivity software like Word and 
PowerPoint, were revealed in the cases of preservice teachers. Hu and Yelland (2017) found when integrating 
technology in instruction, preservice teachers, rather than their students, initiated and directed most of the activities. 
Understanding why teachers choose to use various technologies will advance more breadth in technology use. 


TPACK in Relation to Teachers’ Technology Use 

Earlier literature has extensively examined teachers’ perceived TPACK and its relationships with salient 
cognitive and affective aspects such as teachers’ pedagogical beliefs, self-efficacy, and attitudes towards technology 
(e.g., Crompton, 2015; Tondeur, van Braak, Ertmer, & Ottenbreit-Leftwich, 2017). After recognizing that teachers’ 
perceived TPACK may not aptly reflect their behavioral intention or smoothly translate into their enacted actions in 
the classroom (e.g., Jen et al., 2016), more researchers have begun to unpack teachers’ technology-mediated 
practices to better understand teachers’ TPACK development in the PK-12 environment (e.g., GOmez, 2015; 
Gonzalez & Gonzalez-Ruiz, 2017). Gonzalez and Gonzalez-Ruiz (2017) conducted multiple case studies to explore 
if the behavioral intention of six preservice teachers’ technology-supported mathematical tasks for teaching was 
associated with the TPACK model. Findings showed that behavioral intentions revealed in the participants’ 
explanations of technology-supported tasks was unrelated to the TPACK components. Research that involves more 
participants can expand exploration of teachers’ knowledge when making educational technology planning decisions. 


Pedagogical Reasoning and Action 

Scholars have begun to also consider teachers’ pedagogical reasoning and actions as a conceptual frame 
underlying the technology integration process (Harris & Phillips, 2018; Hofer & Harris, 2019). Developing and 
supporting teachers’ pedagogical reasoning and critical decision-making regarding technology integration in 
classroom teaching is crucial because educational technology options are ever-expanding. Past investigations of how 
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technological knowledge (i.e., TPACK) underlies teacher actions enacted in classrooms has begun to expand beyond 
the ‘what’ and ‘how’ of teachers’ technology use to also consider the ‘why.’ Loughran (2019) asserts “making sense 
of the ‘why’ comes with its own challenges because much of that thinking — the pedagogical reasoning — 
underpinning practice has long been recognized as tacit in nature” (Loughran, p. 4). 

Some studies (Hofer & Harris, 2019; Niess & Gillow-Wiles, 2017) are specifically framed around 
Shulman’s (1987) theory of pedagogical reasoning and action, while others have explored reasoning processes from 
a TPACK conceptualization (Guzey & Roehrig, 2009; Janssen, Knoef, & Lazonder, 2019) or domain-specific 
(Sherman, 2014; Voet & De Wever, 2017) or both (Wilkerson, Andrews, Shaban, Laina, & Gravel, 2016). Of the 
handful of existing studies, most are situated in inservice contexts with a small number of participating teachers 
mostly at the secondary level (e.g., Guzey & Roehrig, 2009; Hofer & Harris, 2019; Niess & Gillow-Wiles, 2017; 
Sherman, 2014; Voet & De Wever, 2017). Two studies involve preservice teachers (Janssen et al., 2019; Wilkerson 
et al., 2016). These researchers contend that studies of pedagogical reasoning will enable better understandings of 
teachers’ knowledge, TPACK or otherwise, within instructional planning and decision-making. 

Studies have identified that teachers’ pedagogical reasoning for technology integration mirrors their 
established pedagogy and knowledge (Guzey & Roehrig, 2009; Sherman, 2014). Sherman’s study of four 
mathematics teachers found that even when teachers had intentions to use technology for high-cognitive activities, 
most often lessons were implemented in low-cognitive ways, which revealed a mismatch between intentions and use. 

The literature also suggests there is a wide range of teachers’ rationalizations underlying their choices of 
instructional technology (Heitink, Voogt, Verplanken, van Braak, & Fisser, 2016; Li, 2014; Voet & De Wever, 
2017). Voet and De Wever found 22 secondary social studies teachers voiced four types of rationales for their 
adoption of technology tools: (a) increasing effectiveness of instruction, (b) connecting to students’ daily lives, (c) 
increasing work efficiency, and (d) complying with peer pressure to use technology. The teachers’ technological 
practices tended to be teacher-centric with little active technology use by students. In Li’s study, teachers articulated 
their use of technology for either attending to personal teaching needs (e.g., professional development or lesson 
preparation) or addressing student learning needs (e.g., enhancing engagement or facilitating understanding). 
Heitink et al. reported that most of teachers’ technology use was to strengthen teaching pedagogy and/or subject 
matter learning. Wilkerson et al. (2016), in a study with preservice teachers, found the teachers valued and used 
technology (a) as a modelling tool, (b) as a way to share student ideas, and (c) as a way to show student ideas. 

These rationales have been identified in studies involving pedagogical reasoning and situated with teachers 
who were planning lessons within specific contexts, such as involving fewer than five pre-introduced technologies 
(Sherman, 2014; Wilkerson et al., 2016); for specific activities, such as inquiry (Guzey & Roehrig, 2009; Wilkerson 
et al., 2016), communication and collaboration (Niess & Gillow-Wiles, 2017), or high-level mathematics thinking 
(Sherman, 2014); or using specific supports or aids for lesson development (Hofer & Harris, 2019; Janssen et al., 
2019, Wilkerson et al., 2016). Through very detailed case studies, past research has begun to identify a range of 
teachers’ reasoning underlying their choices of technology. We had the opportunity to use a larger teacher 
participant pool than past studies to examine teachers’ reasoning for using technologies that they deemed valuable. 

Thus, our study examined a large participant pool of both preservice and inservice teachers, examined 
proposed/enacted techno-lessons, explored the reasoning for technology use in the lessons, and determined the 
requisite TPACK for designing the lessons. Our research questions included: 

1. | What technology-supported teaching and learning do preservice and inservice teachers value most? 
a. Who are the targeted users (students or teachers) in these valued technology-supported activities? 
b. What are the common technology tools involved in the technology supported activities? 
c. How are students or teachers using technology in these technology-supported activities? 
d. How do teachers reason about the value of technology-supported activities? 
2. What TPACK knowledge is reflected in preservice and inservice teachers’ technology-supported activities? 


Method 


This qualitative research study is part of two larger research projects examining technology-related learning 
and integration in one teacher education program and in several middle schools. 


Participants and Data Sources 

Preservice teachers (n=140) enrolled in a university-based certification program for elementary certification 
at a large southwestern U.S. university answered the following questions in a survey administered within two weeks 
of their graduation and teacher certification: 
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e Describe the most valuable learning technology (a learning technology you could not imagine teaching 
without) that you or your students will use in the future, if available? 

e Please explain why your chosen technology (listed above) is so valuable, such as its value to you and 
your students, how you or your students will use it, and what objectives it helps you reach? 

Inservice teachers (n=100) were situated in four different middle schools in a southwestern U.S. state. Saguaro MS 
was located in a rural setting, served a Latinx-majority student population, and 72% of its population was 
economically disadvantaged. Porter MS was an urban school with 50% White student population and 40% 
economically disadvantaged. Walnut MS was suburban with 12% of students economically disadvantaged and 75% 
White. Verona MS was in a rural location and served a student population with 53% economically disadvantaged 
and 57% White. In the teacher survey, these teachers answered: 

e Please list all the technologies (whether you have access to them or not) you feel are required for you 
to have available to assist your students in learning in the subject areas you teach. 

e —_ List the technology and the specific subject matter/content concept(s) it supports. You may be as broad 
or as specific as you like. You can discuss: its value to you and your students, how you or your 
students will use it, and what objectives it helps you reach. 

We examined the inservice teachers as a whole group whose reported practices from the four schools contributed to 
a rich dataset. The open-ended, narrative data allowed deep examination of teachers’ reasoning. A limitation exists: 
we did not observe teachers’ practice and actions in a classroom setting. 


Data Analysis 

The open-ended data was analyzed in a spreadsheet. First, we listed, counted, and grouped each learning 
technology(ies) teachers mentioned into categories used in a common educational technology text. We then 
identified techno-idea chunks that expressed both a learning technology and a use and/or reasoning/value statement. 
For these chunks, we coded for (a) the type of activity (student vs. teacher uses) and (b) the reasoning/value(s). We 
used open-coding using emic and etic coding categories, subcategories, and definitions, while constantly comparing 
our generated codes and coded data to ensure consistency (DeCuir-Gunby, Marshall, & McCulloch, 2011; Miles, 
Huberman, & Saldana, 2014). We also employed multiple coders with code checking between researchers until 
100% agreement on codes was achieved, which enhances trustworthiness (Miles et al., 2014). We then used a 
TPACK knowledge framework culled from the literature (Authors, 2018) to code the requisite knowledge we 
inferred as underlying each teacher’s expressed techno-idea(s) (see Appendix). Finally, we created data displays 
(Miles et al., 2014) to explore and compare patterns in the findings from preservice and inservice teachers. 


Results 
Target Users and Types of Activities within Teachers’ Valued Technology-Supported Lesson Activities 


Preservice teachers’ techno-activities involved more teacher focus (64.6%) versus inservice teachers’ 
activities (44.6%). Conversely, inservice teachers’ techno-activities involved more student focus (55.4%) (Table 1). 


Depictions of teacher use and reasoning within teachers’ valued technology-supported lesson 
activities. Considering the nature of tasks and different aspects of teaching responsibilities, we identified four 
categories of teacher-focused techno-activities (see Table 1): 

e Designing and preparing for lessons—Teachers used technology to prepare for the class ahead of time, 
which included writing lesson plans, researching for course content, creating various instructional 
materials, transferring digital content from home to work. 

e Teaching lessons—Teachers used technology to enhance their teaching when they executed a lesson, 
such as delivering information in multimedia formats, facilitating student-centered activities, showing 
students how to research online, and managing classrooms. 

e Grading and assessment—Teachers used technology for formative and summative assessments and 
grading. 

e Communicating with students and parents—Teachers used technology to communicate with students 
and parents. 

Results showed that the majority of teacher-focused techno-activities that preservice and inservice teachers 

described involved teaching lessons (see Table 1). While preservice and inservice teachers used similar tools across 
all these teacher use categories, inservice teachers valued these activities for their potential to facilitate students’ 
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knowledge development and skills practice; preservice teachers valued the presentational effects that technology 
offered. In the following sections, we compare the common tech tools that preservice and inservice teachers used 
across these different categories of their teaching and the values they anticipated in adopting these techno-activities. 


Table 1. Frequency of Teacher and Student Uses Mentioned in Techno-Activities 


Frequency’: Preservice Frequency: 
Target User and Activities Teachers Inservice Teachers 

n % n % 
TEACHER USES (total) 135 64.6 137 44.6 
Designing and Preparing for Lessons 29 13.9 23 7.5 
Teaching Lessons 93 44.5 95 30.9 
Grading and Assessment 5 2.4 14 4.6 
Communicating with Students and Parents 8 3.8 5 1.6 
STUDENT USES (total) 74 35.4 170 55.4 
Passive Hands-Off Learning (PHOFF) 26 12.4 21 6.8 
Passive Hands-On Learning (PHON) 4 1.9 33 6.3 
Active Hands-On Learning Participation (AHON-P) 30 19.4 70 22.8 
Active Hands-On Learning Creation (AHON-C) 14 6.7 46 15.0 
TOTAL STUDENT AND TEACHER ACTIVITIES 209 100.0 307 100.0 


Note. *n represents the number of mentions within the teachers’ techno-activities; % represents the proportion of 
each group’s total, respectively. 


Designing and preparing for lessons. The most common tools for both preservice and inservice teachers 
were computers, word processing software, and the Internet. Teachers, especially preservice teachers who had less 
classroom experience, valued the Internet because they described it could provide numerous resources for lesson 
planning and creation. Teachers also valued these tools for greater efficiency and flexibility. Several inservice 
teachers mentioned the use of class websites when they designed and prepared for lessons. Teachers usually built 
these sites themselves and added additional tutorials and materials on them, which they could efficiently reuse. They 
valued it as a chance to extend learning for students because students could access these teaching materials at home. 

Teaching lessons. Both preservice and inservice teachers described commonly using the projector, 
presentation software, and computers for in-the-moment teaching. According to the teachers, these tools provided 
greatly enhanced visual effects in their teaching. Some teachers, especially preservice teachers, valued projection 
tools and used videos and other multimedia materials extensively because they felt it engaged student learning to a 
larger extent. Inservice teachers, on the other hand, valued these tools more because they facilitated knowledge-skill 
development and/or practice. Several inservice teachers also mentioned projectors and multimedia were great tools 
to tailor learning towards student needs. 

Grading and assessment. Inservice teachers most commonly used the clicker, while preservice teachers 
most used the spreadsheet. Inservice teachers most valued grading and assessment tools for increasing efficiency 
and monitoring student progress. Preservice teachers, on the other hand, most valued these tools for supporting data 
interpretation and sense making. 

Communicating with students and parents. Classroom websites and email were both the most often used 
communication tools for preservice and inservice teachers. Preservice teachers also applied media creation 
tools/software and word processing for communicating with students and parents. Both preservice and inservice 
teachers valued communication tools for supporting easy communication. 


Depictions of student use and reasoning within teachers’ valued technology-supported lesson 
activities. Our coding of the technology-supported activities teachers valued that targeted students’ use yielded two 
categories, passive and active uses, and were further sub-categorized as follows: 

e Passive hands-off learning (PHOFF)—Technological support for instructional moves and learning 

engagement with subject content, especially to attend to learner needs and variability. 

e Passive hands-on learning (PHON)-Learner use of technology to learn subject content and/or 

revise/practice facts and procedures. 

e Active hands-on learning for participation (AHON-P)-Authentic, learner-driven digital activities and 

learning environments that are typically content-centered and may recognize and accommodate learner 
variability, such as artificial intelligence-sensitive personalized learning. 
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e Active hands-on learning for creation (AHON-C)-Digital learning activities that maximize active, 
deep learning, such as cognitively complex tasks that necessitate creativity, critical thinking, problem 
solving, communication, or collaboration and involve learner agency. 

Of the technology activities described by both preservice and inservice teachers, 26.1% and 37.8%, respectively, 
were coded as active types (see Table 1). While preservice and inservice teachers had nearly identical representation 
of AHON-P activities, preservice teachers described more PHOFF activities, and inservice teachers reported more 
AHON-C activities, in comparison to each other. A prominent pattern revealed that inservice teachers tended to 
value students’ passive and active learning with technology because it led to knowledge acquisition and/or higher- 
order thinking; whereas preservice teachers valued it because they thought the technology supported modes of 
presentation and would engage students. 

Passive hands-off learning involved learners who were hands-off the technology tools, such as when 
students sat in desks and watched, listened, or read digital information presented by teachers. Preservice teachers 
described these activities at three times the frequency of inservice teachers. Common tools described by both 
preservice and inservice teachers included projectors, video, multimodal content, and classroom websites. Yet, 
preservice teachers identified a wider variety of technology tools for PHOFF than inservice teachers, such as the 
Internet, presentation software, and other specific software like spreadsheets and word processors. 

For passive hands-on learning, learners put their hands on technologies to play games to learn content or complete 
subject-related assignments via instructional software. Inservice teachers described PHON activities more often than 
preservice teachers. They also mentioned many more content-specific tech tools such as Compass Learning, digital 
textbook content, and Fastt Math software for PHON as compared to preservice teachers. These technology tools 
were adopted by inservice teachers to support students in developing or practicing their knowledge/skills because 
they thought that these would help to promote students’ knowledge and/or skill acquisition. Preservice teachers 
generally valued students’ passive use of technology to increase their engagement in learning activities. 

A majority of the active uses were activities involving participation (AHON-P) for both preservice and 
inservice teachers. For AHON-P, teachers described students doing subject-specific, technology activities/projects 
such as using LoggerPro to collect science data, conducting online research, creating and showing presentations of 
their work using digital projection, engaging in assessments using clickers, and typing/taking notes. Both groups of 
teachers reasoned that the AHON-P activities enabled students’ higher-order cognitive skills and supported their 
knowledge-skill development and/or practice. Inservice teachers commonly reported using computers and content- 
related software or hardware to promote students’ knowledge acquisition and higher-order cognitive skills. Whereas, 
preservice teachers reasoned that these participatory activities supported students’ use of content/multimedia 
representations, and they cited the internet and projectors as common tools for this. Even among these more hands- 
on, participatory activities, preservice teachers leaned toward showing or demonstrating activities for students. 

Inservice teachers generated more active, creation uses than preservice teachers. Teachers described 
students creating multimodal expressions of learning, such as presentations, graphs, and graphics; publishing writing 
with word processing, blogs, or wikis; and organizing/mapping their thinking and ideas with storyboard or 
mindmapping software. These learning activities positioned students with agency in a range of creative, thinking, 
problem solving, communication, or collaboration decisions. Both preservice and inservice teachers mentioned 
similar technology tools (e.g., media creation software/tools, word processing and presentation software) that 
enabled student creation activities. Both groups of teachers also highly valued student use of technology for creating 
learning products. Not surprisingly, inservice teachers who mentioned AHON-C more often described many more 
additional reasons for creation activities such as supporting their content/multimedia representation, enabling higher- 
order cognitive skills, and fostering collaborative learning, when compared to preservice teachers. 


Teachers’ TPACK Knowledge Underlying the Valued Technology-Supported Lesson Activities 

Overall, both preservice and inservice teachers relied most on TPK (90.5% and 72.3% respectively) to 
justify their reasoning for using certain technologies in their instructional decision-making. High percentages of 
TPK may indicate that both preservice and inservice teachers have a better understanding on how technology may 
be applied for general pedagogical purposes, in comparison to other TPACK knowledge categories. However, 
preservice teachers typically reflected their knowing of TPK while inservice teachers’ rationales mostly on enacting 
their TPK. The difference implies that although preservice teachers had higher reference of TPK in their reasoning, 
they were anticipating what they might do in the future rather than enacted uses described by inservice teachers. We 
suspect preservice teachers’ more limited technology and instructional implementation experiences reduce their 
abilities to have enacted TPK in practice. 

Inservice and preservice teachers’ rationales considered TCK 17.9% and 4.3%, respectively, considered TK 
7.4% and 3.8%, respectively, and TPACK 2.4% and 1.1%, respectively. This may imply that the inservice teachers 
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were given more exposure to technology and content-specific technologies and their possible uses in their school 
contexts than preservice teachers had in the teacher preparation programs. 

Although TPACK represents a synthesized form of knowledge that some argue underlies well-reasoned 
technology integration in teaching, only 3.5% of preservice and inservice teachers included all aspects of TRACK 
knowledge in rationalizing their valued technology-supported instructional activities. 


Discussion 


This investigation is unique because it examined multiple layers related teachers’ technological activities 
for teaching and learning: (a) the technology use activity, (b) the technological tools embedded within the activity, (c) 
the reasoning for use of the techno-activity, and (d) the underlying knowledge teachers used to design such activities. 
This study also was conducted with data from both preservice and inservice teachers, which allowed us to examine 
each as a professional group but also explore if similarities or differences existed based on professional experience. 

The majority of preservice teachers’ techno-activities were teacher-focused, reflecting these teachers’ 
predominant use of technology for in-the-moment instructional use. Some past research trends show teachers 
moving away from using technology for instructional preparation or professional learning toward adopting it more 
often for classroom instructional use (Hu & Yelland, 2017; Ottenbreit-Leftwich et al., 2012). Technologies used 
within lessons, such as preservice teachers’ commonly cited projectors, presentation software, computers, and video, 
provide students exposure to technology, albeit in the hands of the teachers. In fact, preservice teachers’ reasoning 
for using these teacher-focused, techno-activities centered on aspects of presentation/showing, in that they valued 
showing multimedia content information, and felt presentations supported visual needs and enhanced students’ 
engagement. Preservice teachers also generated student-focused techno-activities categorized as passive hands-off 
learning. These activities were almost identical in nature to their teacher-focused teaching lessons activities, tools, 
and rationales with the exception that the descriptions were student-forward and learning-forward (vs. teacher- and 
instructionally-focused). Using technologies for presenting information is common in the literature (Polly, 2014). 

Preservice teachers showed some emergence of active, student-focused, hands-on learning activities, 
especially those for participation. Yet, their common tools, the internet and projectors, and their valuing these 
technologies for supporting students’ representation of content trends again toward presenting/showing, albeit by 
active students. Thus, we did not find it surprising that in our analysis of their knowledge components, their 
descriptions heavily represented TPK, the knowledge of technologies for general pedagogical tasks. Their focus on 
presenting and showing digital information is likely the most common of all general pedagogical strategies. This 
prominent pattern among preservice teachers may reveal gaps in their teacher preparation, such as a lack of 
modelling of student-focused techno-activities and/or content-forward technological activities. 

Inservice teachers in our sample generated more student-focused activities, a trend that differs from the 
extant literature (e.g., Polly, 2014; Pringle, Dawson, & Ritzhaupt, 2015; Tondeur et al., 2016). The inservice 
teachers’ abilities to identify and value student-focused activities, especially those that actively put the technology in 
the hands of the students, aligns with similar emphasis in national plans (Office of Educational Technology, 2016) 
and efforts to increase student agency (Ito et al., 2013; Reich & Ito, 2017) and rebalance digital inequities in which 
students of color or of lower socioeconomic levels tend to have passive technological experiences (Hohlfeld, 
Ritzhaupt, Dawson, & Wilson, 2017). Yet, inservice teachers still described numerous examples of passive, hands- 
on learning, especially reasoning that it supported knowledge or skill practice with games or learning software, 
which are exactly the types of activities other researchers have found to be predominant techno-activities for 
students of color or with lower SES (Hohlfeld et al., 2017; Warschauer & Matuchniak, 2010). While this research 
shows the range of activities and rationales, further research is warranted to examine if teachers’ reasoning for 
technology use changes when considering students with different characteristics. 

Preservice and inservice teachers often expressed different reasoning for similar techno-activity categories. 
Inservice teachers tended to value technology’s aptitude to support students’ cognitive and knowledge development 
or practice, while preservice teachers valued instructional presentation or increasing students’ engagement. This 
pattern of inservice teachers thinking more specifically about students' knowledge development accords with 
inservice teachers’ inclusion of more content-specific tools or activities and, accordingly, a higher percentage of 
TCK they drew upon to think of their activities, as compared with preservice teachers. A range of possible 
explanations for these differences exist and warrant continued future research. For example, these results may reveal 
differences in the teachers’ respective learning experiences: teacher education and professional development, 
respectively. On the other hand, a professional maturation process may explain the differences, in that it just takes 
teachers time in the field as professionals to broaden their techno-activity repertoires to prioritize student-focus. 
Indeed, Gurevich, Stein, and Gorev (2017) who conducted one of the few longitudinal studies of preservice to 
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inservice teacher trajectories, found such an expansion of tools and activities among the novice teachers. 
Alternatively, inservice teachers’ valuing of knowledge and skill development with techno-activities may stem from 
standards-based and high-stakes environments within U.S. schools in which they must ensure their lessons align 
with standards and lead to achievement. It is encouraging that inservice teachers claim to value active, hands-on 
activities for knowledge development because this stance aligns with the research base (National Academies of 
Sciences, Engineering, and Medicine, 2018). 

While the teachers articulate a range of affordances, the teachers’ emic reasoning and values that emerged 
from our analysis reflect teacher perspectives. Sharing these teachers’ reasoning categories with preservice or 
inservice teachers in learning contexts may nudge teachers to consider the value(s) technology offers when 
designing techno-activities and adopting tools, such as during lesson planning design sessions (Janssen et al., 2019). 
Ultimately, teachers may develop deeper justifications for their technological work in the classroom. 

In addition to the aforementioned research directions, we suggest continued examination of teachers’ 
reasoning by subject area teachers to determine how content- or domain-specific technologies within disciplines 
affect reasoning and decision-making, such as studies conducted in music (Crawford, 2010), social history (Voet & 
De Wever, 2017), science (Wilkerson et al., 2016), and mathematics (Niess & Gillow-Wiles, 2017). Such studies 
might also distinguish the instructional lesson outcomes in terms of high or low-level uses (e.g., Sherman, 2014; 
Voet & De Wever, 2017) because categories of teacher reasoning and TPACK do not necessarily infer deeper 
cognitive learning activities in practice. We also suspect that the contexts in which teachers teach will impact their 
valuing of technology, as illustrated by Guzey and Roehrig (2009). Thus, studies can also closely account for a 
school’s technological context as well as the student populations teachers serve while examining the teachers’ 
technological reasoning and action. 
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Abstract 


Popular culture has inspired diverse educational work and research (Guy, 2007, Fink & Foote, 2007, 
Wright & Sandlin, 2009, Johnson, 2018, Heffernan et al, 2018). This project examines how the Great British Bake 
Off (GBBO) might prompt us to rethink assessment design in higher education. Influenced by qualitative TV 
content analysis, we used directed content theory analysis (Hsieh & Shannon, 2015) to examine the 2018 series of 
the GBBO, which comprises of ten one-hour episodes. These GBBO episodes were broken down into their 
performance challenges and were analyzed using Boud and associates’ (2010) Assessment 2020: Seven Propositions 
for Assessment Reform in Higher Education. The analysis reveals that the design of the GBBO is in line with of six 
out of the seven propositions that Boud et al. propose for assessment reform in higher education. In short, there is 
much we could apply to current assessment design, including making assessment central, modelling assessment and 
using open assessment feedback to produce sustainable learners and responsible partners in assessment. 


Introduction 


The type of assessments designed by instructors for productive learning is a subject of much debate. 
Despite advancements made in educational technologies, assessment strategies in higher education appear to largely 
focus on assessments as isolated events, measuring episodic learning. Instructors tend to deploy exams, quizzes, and 
traditional papers to provide some measure of student learning. At times, these assessments have been referred to as 
“disposable assessments”(Wiley, 2017), thrown away and forgotten soon after the assessments are completed. 
Perhaps we need to locate more sources of pedagogical inspiration for assessment design — possibly what Wiley 
refers to as renewable assessments that provide real-world relevance and productive learning. 


Reality television (TV) shows have become a staple in popular culture. In spite of the bad rap they receive, 
we found the performance challenges in The Great British Bake Off (GBBO) Netflix series thought-provoking and 
deserving of a nuanced discussion, particularly on its implications for the design of assessments in higher education. 
In fact, popular culture has inspired diverse educational work and research (Guy, 2007, Fink & Foote, 2007, Wright 
& Sandlin, 2009, Johnson, 2018, Heffernan et al, 2018). So what can popular culture, in the form of the GBBO, 
teach us about adult learning and assessment? 


About The Great British Bake Off 


Despite assessment in education being contentious, watching others being tested makes for good viewing. 
In the past eight years, the world (Ensor, 2013) has been taken by storm by competitive culinary reality shows in 
which amateurs, professionals and even celebrities have their skills and knowledge continuously tested over a 
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number of weeks with progressively difficult challenges. One show which has received a lot of media attention is 
The Great British Bake Off (GBBO) (Conlan, 2015). 

The GBBO first aired in the UK in 2010 and is said to have been inspired by the idea of baking 
competitions at village fétes (Higgins, 2015). In the hour-long show, twelve home bakers compete against each other 
by completing three different challenges: the Signature Challenge, the Technical Challenge, and the Showstopper. 

The Signature bake tests the contestants’ creativity and ability by requiring them to prepare one of their 
tried-and-tested favourite recipes. Bakers present their signature bakes to the judges and receive verbal feedback 
about the appearance and taste. 

In the Technical Challenge, the bakers are put to the test by being given the same recipe on the spot. Here, 
the playing level is uneven, as some of the contestants might not even have heard of the bake they must prepare, 
while others may be well acquainted with either the technique required or the bake itself. The bake is blind tested 
by the judges, then compared against the others (in a think-aloud manner in front of all the contestants), and ranked 
from worst to best. 

The final challenge is the Showstopper. This is where the participants are “able to showcase their depth of 
skill and talent” (Love Productions, 2019). Here, the judges are looking at the contestants’ demonstration of a 
combination of creativity, technical skill, taste and aesthetics. Again, like the Signature bake, the judges give oral 
feedback to each contestant when judging the showstopper. 

After the challenges are completed, the judges deliberate over all the contestants and what they have 
achieved throughout the three challenges. They decide which baker stand out as a star baker and who does not 
perform well enough and will need to be eliminated. 


About Assessment 


Are there insights that learning professionals could implement from this popular show into their 
educational practice? Initially, when watching the GBBO with current assessment trends in mind, e.g. collaborative 
problem-solving (Von Davier et al, 2017), peer assessment (Liu & Carless, 2006), self-assessment (Boud & 
Falchikov, 2007) and learning-oriented assessment (Carless, 2007), it may seem that the show’s assessment design 
is behind the times: the contestants perform alone; the tidbits of feedback they receive while working come from the 
judges as they pass by; the focus seems to be on the end product, which only the judges evaluate; the contestants are 
judged against each other and ranked based on their end product in a type of norm-referencing test (Gipps, 2002). 
On further observation, there appears to be more to this initial view of the GBBO assessment design than meets the 
eye. 


Methods 


Inspired by qualitative TV content analysis, we used directed content theory analysis (Hsieh & Shannon, 
2015) to examine the 2018 series of GBBO. This series comprised of ten one-hour episodes. 


We used Boud et al’s (2010) Seven Propositions for Assessment Reform in Higher Education to code the 
three different challenges in each episode. Although Boud et al’s (ibid) call for assessment reform originates from 
the Australian higher education context, we argue that the underlying principles of his propositions are borderless. 


Table 1. Qualitative Codes (based on Boud et al’s Seven propositions for assessment reform in higher education) 
#1 assessment used to engage students in learning that is productive 

#2 feedback is used to actively improve student learning 

#3 students and teachers become responsible partners in learning and assessment 


#4 students are inducted into the assessment practices and cultures of higher education 
#5 assessment for learning is placed at the center of subject and program design 

#6 assessment for learning is a focus for staff and institutional development 

#7 assessment provides inclusive and trustworthy representation of student 
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Findings 


Table 2. Summary of Findings 
GBBO Overall Design 
#5 Assessment for learning is placed at the center of subject and program. The program is centered around three 
challenges: the Signature Challenge, the Technical Challenge and the Showstopper Challenge. Feedback is given 
during and after each challenge to help the contestants learn and develop their knowledge and skills. 


#1 Assessment is used to engage students in learning that is productive. This show is all about learning that is 
productive. Every challenge aims at contestants being able to produce something. This is an example of 
experiential learning. 


#2 Feedback is used to actively improve student learning. Contestants are exposed to feedback about their own 
work but also to the strengths and weaknesses of other people’s work. 


Signature Challenge Technical Challenge Showstopper Challenge 


#7 Assessment provides inclusive #4 Introduced into the assessment #3 Assessees and assessors 

and trustworthy representations of practice. This identical technical become responsible partners in 

student achievement. Contestants challenge is assessed blindly. learning and assessment. 

demonstrate the knowledge and Contestants are present during the Taking in feedback from the 

abilities they bring with them in this | assessment, where the judges openly judges, contestants build their 

competition. Assessment is based on | discuss and evaluate each contestant’s | own critical thinking abilities 

totality of outcomes. product. This can help develop and are able to independently 
contestants’ evaluative judgement as judge their own work and the 

#2. Feedback is used to actively the assessment process is modelled for | work of other contestants. 

improve student learning. This them, thus helping them to become 

challenge enables the judges to critical evaluators of their own work 

assess later performances ipsatively | and the work of others. In this sense, 

(Hughes, 2017): comparing the the GBBO approach contributes to 

progress of contestants based on their | 48Sessment for learning beyond a given 

initial contribution (this challenge) course (sustainable assessment). 

and the feedback they have received 

throughout to later challenges (the 

showstopper). 


Six out of seven of Boud and associates’ (2010) propositions for assessment reform in higher education 
could be inferred from qualitatively examining the structure and content of series 9 of the GBBO. 

The GBBO is centered around three challenges (or assessments), namely the Signature Challenge, the 
Technical Challenge and the Showstopper Challenge (proposition #5: assessment for learning is placed at the center 
of ...program design). The challenges are not ‘add-ons’ but are considered significant and embedded into the 
program from the very start. 

Throughout the three challenges of the show, proposition #1 is evident: assessment is used to engage 
students in learning that is productive. The challenges are designed to focus on contestants’ learning, which is not 
hampered by any conferment of grades. In addition, the assessed challenges are recognized as learning activities 
which require contestants to engage in appropriate tasks (i.e. baking products), all of which are significant learning 
activities in and of themselves. The baking challenges are organized in an interlinked sequence where challenges 
increase in difficulty (technical knowledge and skills). 

Contestants are inducted into the assessment practice and culture (proposition #4) of culinary arts, not only 
through carefully structured assessments which aid them in making a successful transition, but also through the use 
of the Signature Challenge. This Challenge is an assessment practice which responds to the diverse expectations and 
experiences of those entering the competition. Contestants typically bring their existing unique backgrounds, 
experiences and prior research on the Challenge problem to connect with their new learning activities, in the process 
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making new knowledge connections to existing schema. Through formative and summative feedback at this initial 
stage, rules and expectations around what is needed for success is made accessible to contestants. 

In the Technical Challenge, the contestants are given a task from a professional baker. Identical, minimal 
instructions are provided on the spot. The completed product is assessed blindly and openly by expert judges in front 
of all, and ranked from worst to best. This could also be inferred as inducting participants into the assessment 
practice (proposition #4) of culinary arts. This feedback is used to actively improve “student” learning (proposition 
#2) by being clear and timely in helping them improve on the final task, with the hope of improving the quality of 
their learning and work. 

In fact, it should be noted that proposition #2, where feedback is used to actively improve student learning, 
occurs at all three stages of each episode. During each challenge, students receive informal feedback during their 
baking — not in the form of marks or grades — about how to improve the quality of their work. 

Throughout each episode, the viewer can clearly see how assessor and assessee become responsible 
partners in learning and assessment, as per proposition #3. Taking in feedback from the judges, contestants build 
their own critical thinking abilities and are able to independently judge their own work and the work of other 
contestants. This is particularly evident after the open judging during the Technical Challenge. 

At the end of each episode, one contestant is awarded the “Star Baker” while another is eliminated. This is 
based on the totality of outcomes from the three challenges. Poor or good isolated performances do not determine 
the overall achievement of a contestant in both the single episodes and the whole series. This is in line with Boud 
and associates (2010) proposition #7: assessment provides inclusive and trustworthy representation of student 
achievement. 


Implications of Study 


A study of the GBBO series behooves us to consider the following questions as we design assessments for 
learning in our own practice: 


How will you challenge students technically as well as creatively? 

How will you evaluate student learning along different dimensions and as a whole? 

How will you integrate feedback into formative assessments at different junctures? 

What would a signature challenge for your learners look like? A technical challenge? Or a 
showstopper challenge? 


Finally, is the following GBBO formula a recipe for meaningful and engaging assessment for learning? 
Meaningful engaging assessment = productive learning + learning community + game-like components + 
connect with prior knowledge + open feedback 


Conclusion 


The focus of this discussion is to examine how the GBBO might inspire us to rethink how we design 
assessments in higher education. The GBBO analysis, using Boud et al’s (2010) assessment reform propositions, 
reveals that there’s much to learn from and apply to current assessment design, including the openness to learning 
with others, making assessments sustainable, and beating one’s own personal best. 
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Abstract 


A curated learning experience is a method of organizing vast volumes of web and enterprise-based 
information and presenting it in a structured and significant layout. The study conducted allowed for an 
examination of how educators identify a diverse array of effective applications found in open-access educational 
resources using the Studio Thinking Framework (STF) and the Learning Objects Metadata (LOM) topology. The 
examination resulted in ascertaining ways to identify educational resources that provide student-centric knowledge 
construction and has the potential to inspire learning professionals to curate compelling learning experiences for 


their students. 
Introduction 
Differences exist between empowering learners to evolve and modernizing education (Hosseini, Kees, 


Manderscheid, Réglinger, & Rosemann, 2017). In the 21st century, the use of advanced technical solutions drives 
an increasingly visual culture in virtual applications, film, social media, and advertising (Brantley, 2015). Learners 
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could benefit from modern teaching practices if the educational community embraces the curation of authentic 
learning experiences. Some define a curated learning experience as the method of organizing large volumes of web 
and enterprise-based information and presenting it in a structured and significant layout (Luna-Nevarez & 
McGovern, 2018). 

Inspired content curation has the potential to enhance the student’s learning experience, the teacher’s 
teaching strategy, and to maximize learning outcomes. What is inspired content curation? By inspired, the research 
team contends that educators are capable of locating and incorporating open access resources of extraordinary 
quality in their curricula. With the application of two models for classifying learning experiences and learning 
resources, inspired curation is possible. 

To transition to the disruptive 21*-century environment extensive learning could require more than 
traditional knowledge and skills (Dede, 2010). Also, 21*-century learners will need cognitive skills, and intra- and 
interpersonal skills (Dede, Grotzer, Kamarainen, & Metcalf, 2017). Learners will need to engage, understand, 
persist, apply, and express a response to rapidly changing contexts. More than that, the successful learner will 
reflect and assess solutions (Hetland, Winner, Veenema, & Sheridan, 2013). Learning of this kind reflects 
constructivist pedagogies along with immersive and multi-media learning experiences, which allows personal 
learning and student-centered cognition to occur. 


Research Questions 


The questions posed in this study are: 
1. How could educators use multi-media and immersive resources to curate effective learning 
experiences? 
2. To what extent may students learn cognitive, interpersonal, and intrapersonal skills using open-access 
educational resources from the Internet? 


Theoretical Framework 


The frameworks chosen for the study are the Studio Thinking Framework (STF) (Hetland, et al., 2013) and 
Learning Objects Metadata Topology (Solomou, Pierrakeas, & Kameas, 2015). The foundation of the STF 
consisted of two years of naturalistic observation in studio classes for a goal to develop a work of art. Discovery of 
a second covert or hidden curriculum arose that allowed educators to teach critical cognitive and creative skills 
(Sheridan, 2011). The STF skills encompass Observe, Envision, Express, Engage/Persist, Stretch/Explore, and 
Reflect/Evaluate. Researchers have demonstrated the value of the STF model through its application to virtual 
environments (Steele, Johnston, Lawlor, Smith, & Lamppa, 2018) and over a lifetime of living and working 
(Johnston & Lane, in press). 

This framework gives educators a way to assess open educational resources for 21st-century learning 
experiences that stemmed from naturalistic observations initially with the intent to create meaningful works of art. 
Additionally, the STF aligns to visual education by employing virtual applications and solutions to stimulate new 
educational experiences. Every classroom, home, or office connected to the Internet can obtain a rich assortment of 
educational experiences through available open access to educational resources. Their availability and accessibility 
should inspire learning professionals to select/curate the most effective learning experiences for their students. 

In conjunction with the STF, the researchers chose the Institute of Electrical and Electronics Engineers 
(IEEE) Learning Objects Metadata (LOM) Topology as a basis to assess the selected resources. Secondary and 
post-secondary education learning environments utilize multi-media and immersive resources (Frantiska, 2018). As 
a result, educators can execute this combination through open educational resources. Students can attain 
competencies via several learning objects. Learning objects metadata are neither intangible nor abstract 
applications but the design aims to construct and enhance the learning environment that could have numerous forms 
and purposes (Frantiska, 2018). Hence, learning objects metadata can direct what intangible or abstract applications 
will be required and must be associated with a learner’s applicable educational level (Girvan, 2018). Like many 
frameworks, learning professionals and students alike may characterize learning objects differently. The STF 
model and LOM topology displayed in Figure | includes the description for each. 
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Studio Thinking Framework (STF) 


Institute of Electrical and Electronics Engineers (IEEE) 
Learning Objects Metadata (LOM) Framework 


Visual-based framework 
Focus on visual education by using virtual applications 


Habits/Tenants 
Observe 
Envision 
Express 
Engage/Persist 
Stretch/Explore 
Reflect/Evaluate 


Structural Elements 
Demonstration (Lecture) 


Students-at-work (students to spend in-class time working 
on an assignment, while keeping the classroom focused 
on specific goals) 


Critique (allows students to make connections with habits 
different from those that may have been taught in other 


stages of the class) 


Exhibition (described as an “overarching” structure that 
encompasses the original three) 


Transitions (the time spent transitioning between all other 
structures) 


(Sheridan, 2011) 


Educators can use any entity, digital or non-digital, for 
learning, education, or training 


Educational Metadata Contains 
Information regarding the resource’s learning type 
Exercise 

Simulation 

Questionnaire 

Diagram 

Figure 

Graph 

Index 

Slide 

Table 

Case Study 

Narrative text 

Exam 

Experiment 

Problem statement 
Self-assessment Lecture 


Intended end users 
Teacher 
Student 
Learner 


Instructional context 

Implies the actual context where the learning process 
takes place, and can accept values like “distance 
education,” “face to face learning” and “blended learning 


Educational Metadata Categories/Tenants 
General (groups the general information that describes 


the LO as a whole) 
Technical (provides the technical requirements of the LO) 


Educational (provides the educational requirements or 
pedagogic elements of the LO) 


Annotation (provides educational information about when 
an LO was and by whom) 


Classification (provides information about what 
classification system and LO resides) 
(Solomou, et al., 2015) 


Figure 1. STF Model and LOM Topology 


Population and Sample 


The population for this study consists of educational resources considered open access and readily 
available for use by the public as found on the Internet. The researchers intended to locate a wide variety of open 
access resources based on the following criteria: subject matter, learning resource type, and technology type. The 
researchers chose purposive sampling for this study because this method allowed for an examination of the content 
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made available via online resources (Sheffer & Hunker, 2019). The samples selected for this study included 
examining a multitude of core subject-related content in the Language Arts, Social Sciences, Natural Sciences, 
Math, and Fine and Applied Arts. Additionally, the method allowed the research to appeal to a variety of 
demographic groups. The concept of prominence or recognition, such as nationally recognized TEDtalks, The New 
York Times feature, or validated by other expert testimonials, including most trusted, unique, innovative, or other 
recognition for quality, posited as a requirement for the population and the sample. 

An online search aided in locating potential samples using a series of keywords developed in an earlier 
study (Johnston, Olivas, Steele, Smith, & Bailey, 2017) to locate websites that may feature an immersive and visual 
open access resource that includes a learning experience worthy of curation. The primary keywords used in the 
literature searches included video-based technology, applications, multimedia, and educational levels. Each 
contributor performed a close examination of the content and resources such as language arts, social sciences, 
natural sciences, and math. 

Accessing webpages, such as Pinterest, YouTube, and PBS Learning helped during the search for the most 
technological materials instrumental in understanding the importance and usage of educational resources for 
curating a learning experience using immersive and visual open access resources, thereby allowing for 
triangulation. Accordingly, preferred applications involved those having visuals in the form of demonstrations, 
interactive videos, lectures, or narrative text. The research team did not purchase any examined technologies, 
because one of the criteria consisted of freely available curated resources. The analysis comprised solely of public 
information available online, and thus the team validated no claims in terms of any advertised distinction, subject 
matter, or age group appropriateness. 


Methodology 
Each member of the research team collaboratively documented specific details of at least five free open 
access educational technology applications in an Excel spreadsheet posted in Microsoft Teams resulting in the 
collection of 46 resources for examination, a summary of which is provided in Appendix A. The resulting cross- 
functional matrix displayed a header of columns consisting of: 
e contributor’s name 
title of the application 
retrieval date 
URL of the open access technology 
description of the technology 
academic subject(s) 
educational level(s) 
LOM technology type 
LOM technical data type 
LOM learning resource type 
presence of advertising 
distinction 
contributor’s initial comments following the assessment of the resource 
six categories in the STF model 
rating value for the STF model 
additional comments after rater review 


The cross-functional matrix also included dropdown menus allowing further categorization of the data in 
several columns. For example, the Subject column consisted of a dropdown menu to select the appropriate subjects 
impacted by a technology that included science, math, and the physical sciences. Each contributor completed the 
cross-functional matrix including the STF Habits of Mind (Winner, Goldstein, & Vincent-Lancrin, 2013). The 
contributors identified each Habit of Mind as either Student-Centric, Instructor-Centric, or Not Present in a specific 
application. Also, the contributing researchers included a justification of their decision to label them as such within 
the corresponding cell in the spreadsheet. The contributors added the Rating Value to the matrix later during the 
analysis phase. 
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Data and Analysis 


Using directed content analysis assisted with the interpretation of meaning from the content of textual data, 
hence, permitting adherence to the naturalistic paradigm. With a directed approach, data analysis begins with a 
theory or relevant research findings as guidance for initial codes (Hsieh & Shannon, 2005). Applying a directed 
content analysis on the sample of open educational resources allowed for a classification of the technology based on 
the Institute of Electrical and Electronics Engineers (IEEE) Learning Objects Metadata (LOM) Topology (Solomou, 
et al., 2015) and the Studio Thinking Framework (STF). The Excel document, as an efficient analysis tool, helped 
with tracking, organizing, and ranking resources based on various criteria established by the team aligning to 
attributes of the STF model, the IEEE’s Learning Objects Metadata (LOM) topology, and descriptive information. 

Macros within the Excel document enabled the team to use dropdown menus to categorize each resource 
rather than rely on free-form entry. With Microsoft Teams as the data hub, the team worked collaboratively on the 
Excel document by reviewing each contributed resource and coming to consensus on its fit for the curated 
collection as well as its STF and LOM categorization as a means for identifying emerging themes and trends. The 
research team developed a color legend for the STF model by assigning a specific color for each STF attribute. 
Student-centric attributes received green as the assigned color while teacher-centric attributes received an assigned 
color of yellow. For each open educational resource in the collection, those categories not present received red as 
the assigned color. After assigning a color to the STF attributes, the researchers applied a ranking on a scale 
between 0 and 2 based on the designated color. Red attributes received an assignment of zero, yellow received an 
assignment of one, and green received an assignment of two. 

Totaling each resource’s ranking provided a method to calculate an overall rating that represented the 
overall tendency for the technology to align to the STF model in terms of achievement of the attributes by students 
without guidance from the teacher, that is, distinguishing an inclination as a student-centric or an instructor-centric 
resource. Researchers found that 14 of the reviewed resources had a high STF rating (12, 11, or 10) that suggests a 
student-centric application, while seven had a medium STF rating, suggesting a mixture of student and instructor- 
centric STF attributes, while 26 of those had a low STF rating, suggesting a more instructor-centric attributed 
resource. Figure 2 includes information identifying the characteristics associated with the identified STF ratings 
and the compantion LOM attributes for the reviewed resources. 


Representation of STF/LOM Reviewed Resources 


Studio Arts Thinking Learning Objects 
Framework Metadata Framework 


Student-centric, student Visual, interaction, 
construction of knowledge application-oriented 
occurs independently objects 

Mixed, with some student- Mixture of interactive 
centric activityandsome objects and consumption- 
instructor-centric activity based objects 
Instructor-centric, requires Mostly consumption- 
instructor intervention for based objects such as 
student construction of presentation, text, 
knowledge streaming 


Figure 2. Representation of Reviewed Resources based on STF and LOM Frameworks 
Discussion and Results 


Our first research question is, “How could educators use multi-media and immersive resources to curate 
effective learning experiences?” From our analysis and observations, we contend that by evaluating multimedia 
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and immersive resources using the STF and LOM frameworks, educators can identify and apply student-centered, 
constructivist-oriented learning experiences for their students thereby aiding in the learning process (Muir, Knezek, 
& Christensen, 2004). Students could gain the opportunity to collaborate with peers and engage with the software, 
thereby increasing intrinsic motivation while supporting the needs of the kinesthetic and visual learner (Basaran, 
2016). A profound learning experience could occur because appropriately curated multimedia and immersive 
resources will consist of streaming media, images, text, interactive applications, or include a self-paced activity that 
relies on critical thinking skills (Hosseini et.al., 2017). Multi-media and immersive resources could allow educators 
to shift the learning paradigm and shift a student’s mode of cognition through a more dynamic experience (Girvan, 
2018). 

Presenting knowledge and skills in new ways may challenge students and add value to the learning 
process, perhaps appealing to a diverse learner population. Exposing learners to interactive and vibrant sources of 
information could help learners feel more empowered and increase their level of engagement by exploring such 
curated learning experiences (Solomou et.al., 2015). Additionally, learners could view, summarize, and deduce 
information at their own pace. Conversely, educators could deliver a comprehensive curated experience for 
learners who have learning challenges. The multimedia and immersive resources posited as intuitive, thereby 
reducing the need for assistance when interacting with the programs (Sheffer & Hunker, 2019). 

Our second research question posed is, “To what extent may cognitive, interpersonal, and intrapersonal 
skills be learned using open-access educational resources from the Internet? From our analysis and observations, we 
contend that educators should select multimedia and immersive resources supporting student-centered interaction. 
Through such activity, students have a higher propensity to develop into independent learners who use higher-order 
thinking skills to solve problems and navigate through knowledge sources (Kopzhassarova, Akbayeva, Eskazinova, 
Belgibayeva, & Tazhikeyeva, 2016). Using open-access educational resources from the Internet could allow a 
learner to attain several skills including, but not limited to, synthesizing information, increasing independent 
learning, applying critical thinking and problem-solving skills, and offering more options for a versatile classroom 
learning experience (Wang & Wang, 2011). 

Multimedia and immersive resources may afford learners the opportunity to demonstrate levels of 
understanding through the reflective and evaluative process. Learners could create their own experience if 
developers designed the selected resource for exploration. Using multimedia immersive resources could allow 
learners to acquire cognitive, interpersonal, and intrapersonal skills by selecting resources designed for student 
exploration of an academic topic. According to Clark and Gibb (2006), with the teacher guiding the learning 
experience, learners could acquire such skills by encountering problem-solving scenarios that rely on small group 
interaction. All open access educational resources analyzed included some form of cognitive skill development, 
with cognition described as the totality of mental activities and processes involved in thinking, perceiving, 
understanding, and remembering (Ashcraft, 2005). 

The results of the study support the contention that an identification of multi-media and immersive 
resources on the Internet is achievable and includes information about how educators can locate and deploy them. 
Furthermore, the learning experiences consisting of cognitive, interpersonal, or intrapersonal skills resulting from 
the curated educational applications are shared, as well as the techniques used to identify relevant resources. 


Conclusion 


According to the findings of this research study, many free resources exist on the Internet for educators to 
use to enhance learning. The research team analyzed 46 resources based upon specific criteria and their potential 
usefulness in an educational environment. The standards used to determine the value of these resources included 
educational level, subject, technology type, distinction, presence of marketing advertisement, technical data type, 
learning resource type, and alignment of the technology to the Studio Thinking Framework (STF). Because 
researchers apply their judgment about the efficacy and reusability of digital learning objects (DLOs), using a 
technique to evaluate technology via several measures aids in avoiding bias during the evaluation process of 
technology and could reduce the amount of time required to assess the technology (Basaran, 2016). 

The researchers of this study suggest that open access Internet resources are available for the student to 
interact and experience many of the STF attributes directly, and only require teacher involvement for reflection and 
assessment. These student-centered resources included interactivity and often incorporated multiple paths to the 
learning objective for the student to follow. The Internet resources that did not incorporate means for student 
independent learning required teacher interaction with the program to meet curricular goals. Analysis of the data 
collected incorporated a ranking scale to enable the researchers to rate the extent of student-centrism in relation to 
knowledge construction. The learning resource type on the highest ranked open access online educational programs 
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were interactive. Technical data type paired with the STF framework could lead educators to believe that multi- 
media applications involving multiple forms of interaction receive high ratings for student-centeredness and could 
posit as a good choice for providing students with experiences leading to higher forms of critical thinking. 
Instructors who are inspired to select open educational resources which focus on independent learning and higher- 
ordering thinking provide students with outstanding opportunities to achieve learning objectives. 
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Appendix A 


Title URL Subject Educational Technology Technical Learning STF Rating 
Level Type Data Type Resource Type 
Answer Garden | https://answer; Fine & Applied Middle School, Application, Hypertext Interactive 12 
arden.ch/ Arts High School, Text, Image 
Postsecondary 
Echo360 https://echo36 Math, Social Elementary, Text, Image, Photo, Activity abd 
0.com/higher- Sciences, Social Middle School, Streaming Document, 
ed/ Sciences, High School, Media, Graph, Image, 
Biological & Postsecondary Application Presentation, 
Physical Audio 
Sciences, Recording, 
Language Arts, Video 
Fine & Applied 
Arts 
Flash Card https://www.fla Math, Social Elementary, Text, Image, Presentation, Interactive bP. 
Machine shcardmachine. Sciences, Middle School, Streaming Hypermedia 
com/ Biological & High School, Media, Application, 
Physical Postsecondary Application Interactive 
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Abstract 


Although the process of thinking and learning in the computational practices are important, little research 
has been conducted. This study explored undergraduate students’ patterns and challenges of computational thinking 
practices in an online environment. Qualitative data were collected from Scratch coding journals. The results 
revealed that undergraduates’ reactions towards their successful programming experiences differed. No incremental 
and iterative computational process were identified when sequential directions were provided to complete unfinished 
tasks. Most students tested to see if their programming scripts were working and debugged errors by themselves. 
Few students programmed the tasks using other resources or without any references. The participants created their 
own codes for simpler tasks, whereas they reused and remixed other sources for complex concepts. When they 
mastered computational concepts, the codes associated with the concepts were easily reused and remixed. Practical 
implications were further discussed. 


Introduction 


Our constantly evolving world requires education to prepare students for success in school and life 
experience. Aa literacy is needed for everyone, 4 Cs-—critical thinking, collaboration, communication, and creativity— 
have been identified as the core skills of 21st century learning for every student. Today’s rapid advances in 
computing technology is calling for a new core skill to succeed in the digital age. The ability to expand the horizon 
of human thoughts and mind with computing power has become an integral part of our life and work. Accordingly, 
it has been argued across the globe that thinking computationally is another fundamental skill for everyone not just 
for computer scientists. Computational thinking (CT) has accordingly been gaining increasing attention from 
researchers, educators, and policy makers. 

CT is one of the emerging problem-solving skills that must be acquired by the new generations of students 
to participate in our digital-based world (Barr, Harrison, & Conery, 2011). CT is defined as “the thought processes 
involved in formulating problems and their solutions so that the solutions are represented in a form that can be 
effectively carried out by an information-processing agent” (Wing, 2011, p.1). By using CT as a method for problem 
solving, students can actively participate in the computing world. CT is a broad topic that can be applied in various 
fields to meet the explosive demand for new problem-solving skills. CT can be fostered through the connection of 
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existing standards, including the Common Core State Standards (CCSS) and Next Generation Science Standards 
(NGSS). At the same time, we need to ensure that all students have the equal opportunity to learn a complete set of 
CT through a stand-alone course, as well as being applicable to other fields. 

The rationale for teaching computer science (CS) to students is far beyond career development. CS 
education is about giving more thinking knowledge, skills, and attitudes which help students become active in the 
digital world as active creators rather than passive consumers. Research on embedding CT in K-12 subject areas 
have found that students exposed to CT showed significant improvements in their problem-solving and high-order 
thinking skills (Calao, Moreno-Leon, Correa, & Robles, 2015; Hambrusch, Hoffmann, Korb, Haugan, & Hosking, 
2009). Although, to date, much effort has been devoted to teaching CT competencies in K-12 classrooms, stand- 
alone CS courses are implemented predominantly at the secondary education to teach the CT concepts and practices 
(Yadav, Gretter, Hambrusch, & Sands, 2016). Recent changes in the CS education system have led to college 
students who are currently not receiving such educational benefits. Similarly, most of past research studies have 
been conducted in the contexts of K-12 education, which necessitates research in a more diverse levels of 
educational settings. 

CT has some similarities to other higher-order thinking such as critical thinking, mathematical thinking, 
and engineering thinking. However, the difference is that those thinking frameworks do not include a computational 
process that refer to the domain knowledge, skills, and dispositions in computer science and related fields. The CT 
framework has been built using several concepts of cognitive thinking such as analytical, logical, creative, or 
problem-solving process (Aho, 2012; Barr, Harrison, & Conery, 2011; CSTA & ISTE, 2011; NCR, 2010; Wing, 
2011). Although there is little agreement about what CT encompasses, when it comes to defining CT, three- 
dimension framework has been widely accepted in recent years. According to Brennan and Resnick’s new 
framework for constructing CT (2012), CT involves three key dimensions such as computational concepts (i.e., 
sequences, loops, events, parallelism, conditionals, operators, and data), computational practices (i.e., being 
incremental and iterating, testing and debugging, reusing and remixing, abstracting and modularizing), and 
computational perspectives (expressing, connecting, and questioning). 

Several studies have examined the effectiveness of the CT instructions on student learning in K-12 
classrooms (e.g., Chen, Shen, Barth-Cohen, Jiang, Huang, & Eltoukhy, 2017; Denner, Werner, Campe, & Oritiz, 
2014; Doleck, Bazelais, Lemay, Saxena, & Basnet, 2017; Grover, Pea, & Cooper, 2015; Moreno-Leon, Robles, & 
Roman-Gonzalez, 2015; Roman-Gonzalez, Pérez-Gonzalez, & Jiménez-Fernandez, 2017). Many studies that used 
the Brenna and Resnick’s framework evaluated learning in computational concepts only, while few included 
computational practice and perspectives of the CT constructs. Although the process of thinking and learning in the 
CT practices are important, little research has been carried out in accordance with the practice components presented 
in the framework—being incremental and iterating, testing and debugging, reusing and remixing, abstracting and 
modularizing. Also, Brenna and Resnick (2012) suggested the assessment approaches (e.g., artifact-based interviews 
and design scenarios) to CT practices, but the artifact-based interviews have limitations in terms of the time 
efficiency of collecting data and the dependency of participants’ memory. The pre-determined design scenarios are 
also time consuming and may not be connect to a learners’ interests. Thus, there remains a need for studies that 
further investigate the practical approach to assessing CT practices and provide data offering insight into 
understanding the development of CT practice in while creating a project to solve problems in online higher 
education course. 


Purpose of the Study 


We are still at an early stage regarding the understanding of computational practice development to prepare 
students for 21st century problem-solving. One way to advance in this area is to determine the current status of CT 
practices in block-based programming activities; there is a need to identify particular CT practice’s patterns and 
challenges students have in online programming activities. The first purpose of the study was to explore the patterns 
undergraduate students had while creating their Scratch projects. The second purpose of the study was to investigate 
in which areas undergraduates was having difficulties with CT practice based on the framework proposed by 
Brennan and Resnick (2012). The following four research questions guided the study: 


(a) What patterns and challenges do undergraduates have in the process of being incremental and iterative?, 
(b) What patterns and challenges do undergraduates have in the process of testing and debugging?, 

(c) What patterns and challenges do undergraduates have in the process of reusing and remixing?, and 

(d) What patterns and challenges do undergraduates have in the process of abstraction and modularizing? 
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Theoretical Framework 


Brennan and Resnick’s (2012) CT involves three aspects: computational perspective, computational 
practice, and computational concepts. Among the three dimensions, this study focused on the computational practice 
to uncover the patterns and challenges undergraduate students have in their computational thinking process while 
creating their Scratch projects. The computational practices are defined as the practices a student develop as they 
program. Four main components of practices are being incremental and iterative (e.g., Developing solutions step by 
step), testing and debugging (e.g., Finding strategies for solving problems), reusing and remixing (e.g., Building new 
solutions on existing works or ideas), and abstracting and modularizing (e.g., Modeling complex systems with 
simple elements). The assessment for the computational practices should involve examining processes, which 
provide opportunities to evaluate how their thinking and learning develop over time. 


Method 


Given the nature of our research questions, the study adopted a qualitative research design to yield an in- 
depth and comprehensive analysis. The four research questions focused on exploring the patterns and challenges of 
CT practices from undergraduates. Therefore, through reflective coding journals, we intended to explore not only 
participants’ processes of CT practice in developing their Scratch programming projects but also participants’ 
perceptions of the challenges that novice learners encountered in a context of developing their computational 
practices. 


Participants 

Participants were 96 undergraduate students who were enrolled in the “Computing and Information 
Technology” course from a large public, southwestern university in the spring semester of 2019. The participants 
had enrolled from varied majors, were of various ages, and were both male and female. The students learned a set of 
the core knowledge and skills that shape the landscape of computer science, represent information digitally, and 
create block-based programs to solve problems. 


Context 

The course was completely online and delivered via a web-based learning management system for 
providing course instructions. It focused on a set of fundamental competencies in computer science. The course was 
also designed to provide students with programming experiences using Scratch 3.0 which is a block-based 
programming language developed by MIT Media Lab to makes it easier to create and share their programming 
projects. Scratch is intended to be used in an introductory programming course for people of all ages and across 
disciplines (Resnick et al. 2009), and it offers editors for both online and offline. The participants were asked to 
perform programming tasks using Scratch. Out of 14 modules, there was a total of eight modules related to Scratch 
programming, and the tasks were to complete pre-designed and partially-finished Scratch projects with a set of 
requirements. 


Data collection and analysis 

The coding journal helped capture learners’ thinking process, task completion steps, or their feeling. The 
structural coding journal comprised four reflective questions about their programming process and experiences as 
they created each Scratch programming project. Scratch coding journals asked the participants to share their 
programming experiences with reflective writing in responses to four open-ended questionnaires: (a) Describe how 
you created the quiz show in detail, (b) Describe what worked well during programming, (c) Describe how you 
tested to see if the quiz show was working and how you fixed issues, and (d) Describe how did you adapt a sample 
project to make it your own quiz show. Inductive, thematic analysis was conducted with 85 responses of the 
participants (89% response rate) to the open-ended question in order to obtain deeper insights on the development of 
CT practice. The authors organized the data and then coded Scratch coding journals following the three-step 
guideline of Miles and Huberman (1994): data reduction, data display, and conclusion drawing and verification. 


Findings 
Researchers identified and coded significance statements pertaining to patterns and challenges in the coding 


journal responses. Codes were applied and revised as needed by combining and eliminating codes for parsimony. 
The final number of codes were counted for each category—being incremental and iterating, testing and debugging, 
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reusing and remixing, abstracting and modularizing. According to Scratch coding journal responses, 
undergraduates’ reaction towards their programming experience considerably differed. The results are presented in 
the order of the research questions. 


RQI1. What patterns and challenges do undergraduates have in the process of being incremental and 
iterative? 

In the process of being incremental and iterative, most of students followed the sequences of the task 
requirements in the module. Although no iteration or incremental programming process were identified, a third of 
students designed their own themes or plots before programming with Scratch. Another 30% of students started 
programming after reviewing the sample provided in the module or understanding the task requirements. Because 
CT involves a whole series of process of solving a problem through identifying the problem, designing the project, 
and then coding algorithms, it needs an instructional design where students can go through the thinking and learning 
process for each necessary step. Also, since most problems cannot be solved all at once, it is important to have an 
iterative process to develop solutions with new ideas and approaches. There were limitations in understanding this 
process the task analyzed in this study did not require such iterative process. 


RQ2. What patterns and challenges do undergraduates have in the process of testing and debugging? 

Next, students should deal with various errors that occur during programming. It is important to develop 
strategies for handling problems. In the coding journal, we analyzed various testing and debugging strategies 
developed by learners through trial and error. In the process of programming, parallelism was the most challenging 
concept for testing and debugging, followed by variable, conditional, and operators. 90% of the students tested to 
see if their programming codes were working and debugged errors by themselves. Only 10 % of them used other 
tutorials or consulted with other students. As testing and debugging strategies, instant testing (e.g., testing block by 
block, comparing after with before) and debugging (e.g., fixing an error instantly, comparing with the sample 
project) was the highest frequent approach, followed by persistent approach (e.g., keep testing and debugging until it 
is fixed). The other strategies were consulting with peers, referring other tutorials, and taking notes to fix. 


RQ3. What patterns and challenges do undergraduates have in the process of reusing and remixing? 
Programming based on the work of others is part of computational thinking. Students can improve 
problem-solving skills by make their own based on others’ works, rather than creating something new from scratch. 
In terms of reusing and remixing, 70% of the students adapted the sample project provided in the module to make it 

their own project, and the others programmed the tasks using other resources or without any references. Half of 
them customized appearance changes only (i.e., sprites or backgrounds), and the other half adapted codes (i.e., 
variables, if/else, operators) as reference. Most students followed the overall guidance and direction sequences. 
Students created their own codes (e.g., texts, themes, characters, backgrounds), whereas they reused and remixed 
other sources for a starter or complex concepts. The codes for simple tasks (e.g., changing appearance, adding 
animation effects) is easily reused because they were already familiar with how to code in the previous lessons. 


RQ4. What patterns and challenges do undergraduates have in the process of abstraction and modularizing? 

Lastly, abstracting and modularizing is an important practice in computational thinking by identifying 
general principles and developing problem solutions with simple parts. Students can abstract and modularize their 
solutions by conceptualizing problems in simpler ways and converting the concepts into stacks of code or individual 
sprites. However, unfortunately, we were not able to obtain meaningful insights for the abstraction and 
modularization, from the coding journals because the programming projects in the study were pre-designed. To 
analyze responses for such unidentified practice, an ill-designed or open-ended programming project may be 
required, and also scratch programming artifacts should be analyzed. Further research is needed for a deeper 
understanding of abstraction and modularizing. 


Discussion and Implications 


The findings of the patterns and challenges undergraduates have in CT practice provide meaningful 
implications. One of the interesting findings is that the structure of programming assignments should be changed to 
multiple phases to make it an iterative and incremental process so that students can be more exposed to practice their 
iterative process. The tasks should require learners to build their solution in multiple phases. 

For testing and debugging, it also should provide students with various debugging strategies and resources 
because self-testing and self-debugging would be limited as the difficulty of programming increases. Furthermore, 
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the effective methods of testing and debugging should be provided especially for an online environment. Lastly, 
some detailed guides for customization will be necessary depending on the instructional purposes. For instance, in 
the early stages, students could be asked to make changes on only appearance and move to changes on codes or 
structure, and then to guide them through all changes to the final task. 

For using and remixing, sufficient resources should be provided for learners to explore many resources, 
which could affect more creative and diverse learning outcomes. Reusing and remixing activities offer code reading 
opportunities, but also requires education in ownership and copyright. More importantly, while learning CT through 
programming is important to assess the results of programming, it is also important to provide an opportunity to 
think about why the codes need to be used. Such reasoning questions will ultimately help students improve their 
computational thinking for problem solving. 

This study explored the patterns and challenges undergraduates had during their CT process while creating 
their Scratch projects. We discussed some practical suggestions to provide quality CT education. To improve the 
quality of CT education, all educators should be responsible for how to introduce and effectively teach CT skills in 
their teaching (CSTA & ISTE, 2011; Voogt, Fisser, Good, Mishra, & Yadav, 2015). This study provides educators 
with a better way to design computational thinking activities, especially in an online environment, and also shed new 
light on understanding CT practices. 
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Abstract 


There is an increasing awareness that women are not always given a voice in the halls of Higher Education. 
This has been a recognized problem in the USA for some time. But this problem goes way beyond just the USA. 
Today, 130 million girls are being denied an education. Women need to make their voices heard while men need to 
learn to listen. In the words of Melinda Gates: “A woman with a voice is, by definition, a strong woman.” This 
article is viewed from both an American and a Nigerian perspective. Women and supportive men need to continue to 
share their stories and highlight both the successes and challenges faced. 


Overview 


There is an increasing awareness that women are not always given a voice in the halls of Higher Education. 
This has been a recognized problem in the USA for some time. “In a professional world that has a tradition of the 
‘good old boy’ network, women have long fought for recognition in the field of educational technology” 
(Donaldson, 2016, p. vii). But this problem goes way beyond just the USA. Today, 130 million girls are being 
denied an education. One brave advocate for females is the courageous young girl, Malala Yousafzai, who risks her 
life to advocate for education for all girls (Malala Fund, 2019). As a winner of the 2014 Nobel Peace Prize, Malala 
is definitely a voice that is being heard. 
In Nigeria, young girls continue to fight for their education. The most recently available adult literacy rate 
(2015) is only 59.6% in the most populated country in Africa (UNESCO, 2019). There is a Nigerian tradition of 
considering boys as the priority gender to receive education. Many women have shared how they have gone against 
their fathers, husbands, and communities to fight for their education and the goal of an academic place within the 
country. This paper will share from a diversity of perspectives individual women’s stories of their personal journeys 
in the face of daunting obstacles. 
Women need to make their voices heard while men need to learn to listen. In the words of Melinda Gates: 
“A woman with a voice is, by definition, a strong woman.” Attendees at a recent Association for Digital Education 
Communications Technology (ADECT) conference in Abuja, Nigeria in May of 2019 shared many stories when 
asked to make known their experiences. Personal stories where shared from experiences in the USA, Namibia, and 
Nigeria. According to the Malala Fund website (2019) the following are some of the reasons and examples of why 
approximately 130 million girls and women are denied an education: 
e ~=Child labour: Zainab received less than one cent per day stitching footballs, but she knew she could 
achieve more for herself and her family by finishing school. 
e Early marriage: When the day came for 14-year-old Najlaa to be married, she felt her dreams of 
finishing school slipping away. So she ran away from home — still in her wedding dress. 
e Conflict: By age 17, Zaynab was a refugee of three wars — in Yemen, Somalia and Egypt — and had 
been out of school for two years. 
e Cost: Like many girls in Nigeria, expensive school tuition fees put Amina’s education and future at 
risk. 
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e Gender bias: In Pakistan, many girls like Nayab drop out after grade 5 because their parents believe it 
is a waste of money to send a daughter to secondary school. 

e Health: In Ecuador, Daniela graduated secondary school, becoming one of seven girls in her class with 
a high school diploma — and a child. 

e Natural disasters: When an earthquake struck Sydney’s village and damaged her home in Oaxaca, 
Mexico, she worried about falling behind in her studies. 

e = Poor quality: Living in a refugee camp in Jordan, the only class available to 12-year-old Rehma is for 
five and six-year-olds. 

e Poor quality: In Syria, Rehma was a promising student, but today in her one-tent school, she’s 
repeating lessons she learned years ago - the alphabet, numbers, the names of colors and animals. 
Rehma says her dream is to graduate, but if no higher grades are available to her, she never will. 


Ana’s Perspective: Voices Shared from the USA 


The historical situation often was not as extreme as the above examples but did include many gender-based 
challenges. Women played an early and important role in the history of technology. Ada Lovelace, the daughter of 
English poet Lord Byron, expanded on Charles Babbage's theoretical device in the mid-1800s, creating the first 
computer program. During WWII, women took over computer programming while the majority of men served in 
military roles. The women behind the popular film, Hidden Figures, got the Apollo 11 to the moon and back during 
the 1960s. Admiral Grace Hopper was a computer scientist and trail blazed the transition to the sophisticated 
computer programming languages that we still use today. She also was the one to coin the term “bug” when a moth 
got into the computer hardware and caused a malfunction. 

Dr. Jean Kilbourne was one of my earliest influences in realizing the inequality that surrounds women in 
our society through the advertising medium. There are now 4 editions of her award-winning documentary, Killing 
US Softly. She speaks very loudly about how Madison Avenue has perpetuated the myth that women are to be seen 
and not heard. Another advocate for females is the courageous young girl, Malala Yousafzai who was awarded a 
Nobel Peace Prize for her voice in advocating education for all girls. 

To provide my own foundation on the subject of women in leadership positions, I will share a bit of my 
own story. I was raised in the Eisenhower beige world of the 1950s, As the first wave of the Baby Boomers our 
options as women were limited. The only future choices I was given included housewife, secretary, nurse, or 
elementary school teacher. I remember my mother insisting that I take a typing course instead of physics. She 
wanted to make sure that I had the necessary clerical skills while insisting I was not smart enough for advanced 
science classes. 

While in college, I was required to take a course on the History of American Education. I rebelled when I 
discovered that the only women mentioned in the text book included the wife of the first president of Harvard and 
Maria Montessori who was Italian. Women did play a very important role in the early American education though 
they were required to be single, church going, and of high moral standards. 

Prior to returning to college to finish my degrees in my mid-40s, I was a computer programmer. I always 
was uneasy with the very masculine terms that described problems: programs crashing, killing a program, or 
dealing with a virus. The accepted terms for what I did in designing early web pages also were not comfortable, I 
was neither a Web Master or Mistress. My final chosen label was Cyber Goddess. It was not always easy being the 
only women in an office of men who were surprised when I refused to fetch them their morning coffee. 

Because I believe that the voices of women in our field are not always heard, I edited the book, Women’s 
Voices in the Field of Educational Technology, Our Journeys (2016). I asked leaders in our profession, men and 
women, from a total of 5 countries to share their stories. Through this effort I learned that women have to fight for 
their voices while men’s voices are simply a birthright. 

Today many women have just begun to raise their voices to share hidden experiences of sexual harassment 
and assault. One of 4 or 5 college women have been sexually assaulted per a recent RAINN report (2019). While 
21% of transgender, genderqueer, nonconforming college students have been sexually assaulted, A third of rape 
victims contemplate suicide and 13% succeed. The fact that women and nonconforming students are coming 
forward to share what many consider their ugly secrets is bringing a major problem to light. 

There exists a gender imbalance in attendance at higher educational institutions in the US. The male 
dominance started to shift downward during the 1970s and the 1990s saw the number of women (71%) increase 
substantially to outpace the 20-year static figure for men (61%), Other countries have other representations (Lopez 
& Gonzalez-Barrera, 2014). 
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Women in the USA are less likely than men to achieve tenure. “While women held nearly half (48.9%) of 
all tenure-track positions in 2015, they held just 38.4% of tenured positions (Catalyst, 2017). The Catalyst report 
goes on to state: 

e While women represent over half (51.5%) of Assistant Professors and are near parity (44.9%) among 

Associate Professors, they accounted for less than a third (32.4%) of Professors in 2015. 

¢ Women held over half (57.0%) of all instructor positions, among the lowest ranking positions in 

academia 

e = 22.1% of women faculty are in non-tenure-track positions, compared to 16.8% of men faculty. 

e =At all categories of institutions, full professors who are women earned on average $98,524 a year 

compared to $104,493 for their male colleagues in 2016-2017: only 94.3% of what men earned. 
(Catalyst, 2017, p. 1) 


Within our own International organization, AECT, there have been some interesting trends. Women in the 
US did not get the vote until 1920, just 3 years before AECT was birthed. Since that time only 17 women have 
served as an AECT President. There was even a gap of 32 years with only male presidents. With the current 
membership ratio of 54% women the hope is that this trend is being reversed. In fact, the 2019 slate of members 
running for the next president contained only women (Doyle, 2016). 


Felicia’s Perspective: A Voice Shared from Nigeria 


I have my higher education degree from Auchi Polytechnic, University of Portharcourt, and University of 
Nigeria, Nsuka, Nigeria. I am currently a lecturer, a researcher, a passionate educationist, with a specialty in 
educational administrator, educational management, and educational technology. 

Illiteracy can be equated to sickness of the body and blindness was what my mother always told me as a 
child. She informed me that an educated woman is an empowered woman. She also told me to ensure that I worked 
hard as a woman. These wise words were always ringing in my heart and made me work even harder. I went through 
school as a shining star even though there was no money to support my education. I was considered to be a brilliant 
student in my secondary school and every other student ran to me to help them with their academic difficulties. In 
those days in school, most times | did not have the required text books. My fellow students who had the textbooks 
were always bringing their books to me for coaching and more explanations. I taught them and afterwards used the 
books myself. My exam performances were always outstanding and far better than the students with the books I 
borrowed and taught. I eventually became the talk of the town and was popularly called SP as 1 was the Senior 
Prefect. As a child I also went by the nickname Doctor since that is what I wanted in my future. As a teenager I 
returned to our rural village from the city of Lagos. The culture questioned education for daughters and I was pulled 
out of school to work the farm. My teachers influenced my father into continuing my education. Around me 
individuals kept telling me to marry instead of furthering my education. With the support of my father, I came to 
realize that I was blessed with an inner drive and courage 

When I was ready for higher education. I was first admitted to study Agricultural economy at the 
University of Ibadan but was denied going because my father was wrongly advised. My father took me to a lecturer 
from my village who then admitted me to study secretarial studies at Auchi Polytechnic. Surviving school was not 
easy. I graduated and started working. I knew within me that I wanted to get to the top academically but for financial 
reasons I had to first find a Job. 

In time I got married after a few years. Faced with married life, I saw my academic dream slipping away. I 
had my first child and then with a great deal of courage and a supportive husband, I went back to school. Eventually 
I graduated, and then went back to school for my Masters. Today, I am happy to have earned the title of doctor of 
philosophy with a specialization in Educational Administration and Planning. Our culture did not encourage me in 
my desired medical degree but I have achieved my dream with my PhD transitioning from medical to academic 
doctor 

Even though I did not receive by PhD until 2017, I have accomplished many milestones along the way. I 
established a private primary school in a country where literacy is not always the focus for young girls. My research 
has been presented at international conferences at Harvard and Indiana University. I was the founder of the 
Association of Digital Education Communication Technology and organized the initial AECT - ADECT 
professional conference. Through this affiliation, I have created lasting collegial relations with key AECT scholars. 
At this point in my journey I’m excitedly waiting to see what the future holds. 
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Nigerian Past and Present Voices 


Nigeria is currently the largest African nation in terms of population and within a few years will be third in 
the world behind China and India. The latest available figures show 59.4% female literacy and 74.4% male literacy. 
As of 2018, the illiteracy rate for 15-24-years-old was 9,364,626 with males at 3,509,338 and females at 5,855,288. 
Among those 15 years and older (41,763,792), 15,904,134 were male and 25,859,658 were female (UNESCO 
Institute for Statistics: Nigeria, 2019). 

The culture continues to encourage males in education while women and young girls are given limited 
options. In 1920, only 7.7 % of Nigeria's college students were female By 2009, number was increased to 45 %. In 
spite of the high percentage pursuing higher education, females constitute just 20.3% of lecturers in Nigerian 
universities. This is a country where the terrorist Boko Haram continues to thrive while using kidnapping and 
suppression to discourage education or equality for girls. 

When researching the noted women of Nigeria, the list included the following names: 

e = Prof. Adetoun Ogunseye — 1‘ female professor 

e Grace Alele-Williams — 1‘ female Vice- Chancellor and 1“ female PhD 

e Funmilayo Ransom-Kuti — teacher, activist, founder of the Nigerian Women’s Union 

e Kuforiji-Olubi — Headmistress at 19 who became the first female graduate from the male-only 
Chartered Accountant of England and Wales. She was the also the first female graduate of ICAN 


There are many identified societal benefits in educating women. These include a direct correlation with 
improved health and increased quality of life. Educated women are more likely to seek proper medical care both for 
themselves — especially maternal care — and their children. Higher rates of female education correspond with 
lower HIV and STD rates. Educated Nigerian women are less likely to get married or give birth as teenagers which 
improves society as a whole. These women are also more likely to hold stable jobs, less likely to be in poverty, and 
more likely to contribute to the overall economy (King & Hill, 1993). 


Discussion on Future Opportunities 


Women’s voices are starting to be heard. More women are assuming prominent leadership roles in higher 
ed, the corporate world, and politics. Programs and professional organizations are in place at many universities to 
nurture future women leaders. And the generation entering college appear to not support the mantra: well we’ve 
always done it that way. 

As many of us have been mentored, we have learned how important it is to support emerging scholars and 
colleagues. We value the act of reaching a hand back to those who follow. Leadership is taught by example, not by 
words. So how do we take this forward? Women need to make our voices heard while men and administrators need 
to learn to listen. A proven approach to support and to encourage emerging scholars and leaders is to: 

e ~— Be an intentional mentor 
Actively invest in a mentee’s success 
Purposefully build mentees’ confidence 
Assure early successes 
Promote mentees’ strengths to others 
Treat mentees as your colleagues 
Invite mentees to work alongside you 
(co-teaching, co-writing, co-serving) 


e 

The quote that begins the Women’s Voices book is by Alan Rickman: “We need to tell stories to each other 
about who we are, why we are, where we come from, and what might be possible” (Donaldson, 2016. p. vii). 
Women and supportive men need to continue to share their stories and highlight both the successes and challenges 
faced. As our voices are heard, let us be the chorus that supports all those whom blaze their own future paths at all 
international levels. 
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Abstract 


This study introduces an effort in the direction of an effective dynamic mini-courseware design during the 
practice of learners from multidisciplinary in higher education including several areas of professions such as 
learning science, computing, information science, artistic design, curriculum development, management, marketing, 
and law. The nature of a heterogeneity in a multidisciplinary team contributes pecific mental cognition and expertise 
when achieving socio-emotion and task over a social representation in a common representational space (Gloor, 
2006; Vlad - Petre Glaveanu; 2014). The finding suggests heterogeneity of teamwork is not limited to a boundary of 
location, enriched by a virtual multidisciplinary members in a Design Thinking process encompassing; Diverse 
inspiration, Sharing empathy, Generate idea, Competing prototype/solutions, and Pilot-testing via a widespread of 
geography and time differences. A promising result empowering the VR multidisciplinary team for a “power of 
thinking” through a communication technology that nurturing an iterative process of thinking with social-ethical 
practices with the end-users’ empathy sharing. 


Introduction 


Aside from a professional skill, a demanding of non-routine analytical skills including problem-solving, 
communication, teamwork, creativity, and entrepreneurship are acquired to be the essential skillsets for all careers 
that ensure the employability during the disruptive change of the 21st century. In parallel with the country goal that 
aimed for uniting diverse individuals into a productive quality citizen of the country; Digital Education, a promising 
path that empower college students and life-long learners, intertwines content with technology encompassing 
approaches of technical perspective, learning perspective, collaboration in a social-ethical perspective. Such the 
digital education system falls under the lead of Educational Technology profession that aimed at facilitating a 
learning capacity by the process of creating, using, and managing technological enhanced environments. Digital 
Education system in a technical term means a system for teaching, learning and evaluation with some basic 
automatic features such as attendance monitoring, media audio visual lesson editing, online practice/exams, 
individual assessments, centralized data storage, etc. The system in a holistic approach allows an involvement of 
practitioner/professions in a broad range of disciplines and members from schools, colleges, universities, non-formal 
institutes, and private sectors. The Digital Education system includes a total management; a plan for a quality 
assurance, strategic teaching, learning engagement, monitoring, evaluation, financial and benefit control. The digital 
education system urge an effective team to work for the success; integrating knowledge and methods from different 
disciplines, using real synthesis of approaches. Several researches addressed issues of the content in the courseware 
product that focused on value for the learners and the development methods with multi-disciplinary team. A few 
concerns on a courseware development disciplines cooperate for the best of the quality of interactive product for 
learners. A courseware design are usually text, graphics, animations, video, and audio wherever it is functional or 
otherwise suitable. The level of interactivity is usually restricted to interactive forms available through technical 
aspects. A simple technical function somehow could reach an effective learning outcome when applying with a 
decent design of instruction/presentation. In the digital learning system from teaching/learning perspectives; 
content, technology, and delivering are holistically in concerns, as the fast speed of technological change disrupt 
shortening the life-cycle of knowledge, delivering mode, and how people learn. Multi-disciplinary team in a 
courseware development disciplines is likely to result in a dynamic and raise the level of learners’ awareness in all 
aspects, especially when employing users’ centered approach of Design Thinking. 
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Mini-courseware 


Contemporary learning has been enriched by state of the art technology namely mobile Learning (m- 
Learning) and ubiquitous Learning (u-Learning). The learning via mobile is attracting considerable interest in the 
fields of professional learning and work-based education, as its capacity of just-in-time and on the job upskilling 
people at work. Some design approaches are inclined to merely adjust the content to fit the screens of mobile 
devices, still employing behavioural and cognitive paradigms. Numbers of research enrich mobile learning 
strategies that accommodate creation and sharing of content; learning across formal, non-formal settings and at 
work. Psychological implications are broaden to situated, socio-cognitive, cultural, multimodal and constructivist 
educational paradigms (Pimmer, C. & Pachler, N., 2008). The new generation of learning system via mobile device 
is precise, brief, effective, and can dynamically interact with learners’ individual profiles, the term is coined “mini- 
courseware” (Li, Y., Guo, H., Gao, G., Huang, R., & Cheng, X., 2009) . Largely known as “micro-learning”, skilled 
based learning approach, a courseware appears in relatively small learning units and short-term-focused strategies. 
The instructional strategy highlights a microlearning process which is the design to gain learners’ interaction on the 
micro-content, learning time and cost effective, specific topic/unit, small part of curriculum, knowledge/skill 
elements, integrate/situated process, multi and mediated media form, active/pragmatist approach. For more 
example, Uptodate (known in a medical care) an evidence-based learning resource, a contemporary learning ground 
that has been enriched by state of the art technology of Big Data, Micro learning, Mini-courseware and mobile 
Learning. This type of courseware design has been integrated into the mainstream of education and training. This 
design direction has caused the field of Educational Technologist from scope of integrating media and learning to 
face an evolutionary challenge since this new demand of learning must effectively correspond to an individual 
learner’s needs in an appropriate manner and at the right time, within a disruptive but short life-cycle of demanding 
knowledge and technology. The scope is broadened to other professions as Information Technologist, Computer 
Engineering, Business, Law, etc. For the teaching and learning part, a quality content/courseware is gained not only 
from technical or pedagogical part, but other areas of the contribution and precaution to the issues that will 
encounter along the development and implementing the learning courseware. A Multidisciplinary team, individuals 
from different disciplines work together contribute their disciplinary knowledge for success. Team, independent 
individuals effort collectively to achieve a common goal or task with an effective process, is beneficial to problem 
solving that brought various perspectives as claimed to meet effective possible solutions with components of 
combination of team, communication, coordination, motivation, and relationships of the team (Paulus, P, 2000; 
Hoegl, Martin & Hans Georg Gemuenden, 2001; Ilgen, D. & Hollenbeck, J., 2004). Team member could offer 
their unique knowledge and ability for team output and motivated individuals’ ambitious to improve their 
performance (Katzenbach, J.&; Douglas, S., 2015). Recent research on teamwork enriched by various expertises 
could gain a group wisdom through a process of conceptualizing knowledge, virtue and emotional feelings, and 
operating in a practical action with the group judgment and ethical decision (Akgiin, A.E, 2019). 


Virtual Multidisciplinary Team 


Team consists of a group of people with needed skills to accomplish a particular task. Working as a team 
may require a process to implement a team training to promote a shared goal, strategies, and resources, and 
participation as well. Not only a requirement in a workforce, some research propose several approaches for in 
higher education students training programs; for example, using an authentic context such as a simulation game 
development to enhance teamwork skills. The study employed a game-based software development to be an activity 
in enhancing a teamwork skill found an issue of a lack of acceptable group assessment (Sereti, et.al., 2019; Iacob & 
Faily, 2019). Researchers propose an evaluation providing immediate feedback allowing a transfer of learning 
which the skills will eventually be modified to a similar one. 

Team, a group of individuals together working from different geographic locations synchronized and relay, 
could effectively perform in a geographically dispersion relying on communication technology, as known a virtual 
team (Nader, A., Shamsuddin, A.& Zahari, T, 2009; Zahari, T Nevogt, D., 2013). 

A few process of virtual team is found; Lipnack, J. & Stamps, J. (1999) a virtual team, consisting of three 
major components; people, process and connection, promotes a dynamic and diverse group and a common team 
structure might not be a favor of the team. Nevogt, D. (2013) proposes a practical process and technological tools 
for virtual team. The guidelines of the virtual team are 1) Goal settings and valid explanation 2) clear and precise 
notification for tasks and its priority with deadlines and publish for member to be follow and monitor the progress 
3) Follow-up based on the agreement tasks and timeline as a group not individual communication 4)Allow for 
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reasonable iterative process to develop the solutions 5) Technology mediates the process such as an online project 
blueprint--flowcharts, text-written, mind-maps, video; group and task internal communication; expert connections. 

Research review on a virtual team consists of 4 major components: members, social-emotion, process, and 
technology. The heterogeneous group of cognitive specific from multi-discipline joined a virtual team could bring 
an exceptional solution. Socio-emotion issues take a major part in this diverse cognitive sharing. Some research 
supports the face-to-face meeting could empower virtual team to share a mental model and lessen the diverse 
cultural difference that could obstacle an effective communication. (Suchan; Hayzak, 2001). Take process via 
Computer-Mediated Communication (CMC) is highlighted as a success factor of virtual team. Technology takes 
roles in keeping members in following up task process even more crucial than a face-to-face teamwork. Members 
could be in a silence without noticing the progress to the group and that cause an ambiguity and eventually become 
problematic. Group coordination and task-technology-structure fit could keep up the virtual team forward the work. 
Finally, collaborative technical aspect takes an essential effects on team 's performance in following the taks and the 
process and that contributing to the work and the satisfaction of belonging to the team. Technology could provide a 
nurturing climate of which members could build a trust and cohesion of the team using a dialog or conversation in 
small group to share positive or release a doubtful sharing mental models during a virtual team norm building. An 
extensive reliance on communication technology anyhow could reduce effective of virtual team. Some research and 
practices apply a reward system, creating trust, and communication culture for team cooperation, also periodical 
face-to-face meetings to form relationships and being a vehicle to coordinate activities. In addition, teamwork 
needs an ongoing practice to maintain and promote the development of a shared mental model of multidisciplinary 
participation with team resources and goals. In addition, immediate feedback on learning outcomes could facilitate 
the transfer of learning during the practices. 

A social hierarchical structure appears in an organization or group that a top position or senior tends to 
become authoritative and individualism causing trouble in engaging collaborative work. Also, some task could be 
appropriate for individual work that also could lead to a conflict and damage to the teamwork. Using teamwork in a 
curriculum, senior project of software engineering team exposed students as complex learning in a professional 
work environment, and that students are well aware of the commitment before commencing the group project. 
However, unequal amount of individual’s work was in cautious that could create conflict and lead to lower levels of 
performance (Iacob, C. & Faily, S. 2019). 


Design Thinking 


Learning design, is a practice of matching learning with constructive instruction enhanced by technology. 
Several research studies found Design Thinking to be a powerful method providing a solution-based approach to 
complex problem solving that addresses ill-defined teaching and learning problems. Design Thinking is a technique 
originated in 1960’s in Engineering field for an innovative production. Later, 1970’s the design became a subject in 
replace to traditional arts and crafts for secondary school in the U.K., and gradually link to technology study. 
Design Thinking, in 2003 has been introduced at the university programs in business and innovation areas 
(Lockwood, T., 2010). 

The process of Design Think is to understand the learners needs, re-framing the problem in a learners- 
centric way, through brainstorming techniques, prototyping and testing (d.school at Stanford University, 2010; 
Adams, C., & Nash, J. B., 2016; Brenner, W., Uebernickel, F., & Abrell, T., 2016). Typically, the process primarily 
includes a technique to gain an empathic understanding of the problem from the learners’ insight by engaging and 
empathizing with learners’ based on their experiences and motivations in the learners’ physical environment. A 
wide and open view, aside from courseware designer’s personal assumption, will clearly give insight to the learners’ 
status of the problems. The next stage, the gathered insightful information is analyzed and synthesized to define a 
core problem, and through teamwork will validate a defining task of a problem statement. Eventually, the solutions 
come through supported by a teaching/learning mechanism that is well enhanced by technological features and 
functions. Later, in a prototype phrase, the design team scales down versions to investigate the problem solutions 
from the ideate stage. The team will perform a validity of the test with a variety of small group sampling 
techniques. Thorough this stage of the testing, the details of the prototype are investigated based on learners’ 
reaction whether to improve, accept, or reject on the basis of learners’ reactions. The best solutions will tune in this 
pre-experimental stage. The design thinking process with a teamwork should give an innovative solution resulted 
from a multiple loops and in different angles. 

Design Thinking is represented with three iterative stages: inspiration, ideation and implementation 
(Brown, 2008). Inspiration, motivating the search for solution, generated through interactions with target clients. 
Ideation is the process of generating, developing and testing various ideas leading to solutions. Finally, the 
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Implementation stage of design thinking plans a path to develop and run the selected concept. Review the 
development process of Design Thinking found common cores accordingly (Brown, 2009; Brown, T. & Wyatt, J. , 
2010; Stickdorn & Schneider, 2010; Coakley, Roberto, & Segovis, 2014; Glen, Suciu, Baughn & Anson, 2015; 
Gachago, Morkel, Hitge, van Zyl & Ivala, 2017): 1) Inspiration, understanding the problems and opportunities; 
2)Empathy, Users’ center deep analysis from users’ physical and emotional experiences; 3) Ideation, alternate 
between divergent and convergent thinking through a brainstorming and think outside the box 4) prototyping & 
implementation, the idea turn into a concrete product/process and iteratively evaluate and refine. 

Design thinking is found its largely used in software development organizations as a tool to follow a 
human-centered approach. Design Thinking also is claimed as a tool for undergraduate students’ creative 
skills,problem solving, communication, as well as collaborative work, as known social and cognitive skills. Much 
more than that, Design Thinking, empowers students to train acquiring empathy towards others. (Valentim, N. M. 
C., Silva, W., & Conte, T., 2017) Brown (2008) uses a “persona”, a character of users, to be an action to describe 
detail clients’ characteristics, typically, represented in a textual form and photo, to motivate teams into thinking 
about users during the design process, making efficient design decisions. During the stage of Empathy; four areas 
accessing the users 1) Think and Feel: what users’ perceive the issues/things 2) Hear: how the environment 
influences the users 3) See: what the users see in their environment 4) Say and Do: what the user says and how users 
behave in public. (Jahnke, I., Lee, Y., Pham, M., He, H., & Austin, L., 2019) 

This research is aimed to provide guidelines of building up a virtual multidisciplinary team based on a 
design thinking process to achieve the target of a dynamic mini-courseware application. The study is conducted by 
a systematic review of research and best practices of design thinking process with the use of communication 
technology 


Purposes 
This empirical study aimed at outlining the process of building up a virtual multidisciplinary teamwork 
within a Design Thinking process. The study also explore tools and process in bridging communication among the 
multidisciplinary team while performing dynamic mini-courseware design. The main research questions are: 
e What are the process and communication tools in the Design Thinking for a virtual multidisciplinary team? 


e How the multidisciplinary team performed in the process of Design Thinking? 
Methods 


Building a virtual multidisciplinary team with Design Thinking process is a qualitative method in its nature. 

1. the study employs a qualitative data collection using research and best practices review, analyze, and 
synthesize the process of Design thinking. In a matrix format, the synthesize process of Design Thinking 
is crossed analysis with a multidisciplinary team practice. 

2. The result of the analysis of a virtual multidisciplinary teamwork with Design Thinking process is 
matched with a categorized technology, derived from a short-listed expert reviews, literature, and a 
software survey. 

3. The study apply an empathy technique interview to understand the underlying opinions of some potential 
members that diving deeper into possible problems while making a team to complete the mini-courseware 
design. The data is gathered to identify the solutions with three experts’ opinions. The process of Design 
Thinking in a virtual multi-disciplinary team was adjusted. 

4. Eventually, the process of Design Thinking of the virtual multidisciplinary team via the technology 
toolkits was piloted. The researchers’ participated and observed, followed by a semi-structured interview. 


Results 


Design Thinking with Multidisciplinary teamwork process consists of five stages namely: Diverse inspiration, 
Sharing Empathy, Generate Idea, Compete Prototype, and Pilot/Testing. 
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EMPOWERING MULTI-DISCIPLINARY TEAM 
S3D1L Ved WWDIHLI-WID0S 


Figure 1. Model of building a virtual multidisciplinary team with design thinking 


1. Diverse inspiration: 

The Diverse group inspire: This stage emphasizes an awareness of the diversity of the group, welcome and 
persuade the group to the ultimate outcome. A clear and precise goal is announced on the homepage of the team. 
The group task is analyzed and along with the sub-teams’ responsibility and timeline must be a group 
communication. This stage consists of 3 steps 1) Represents heterogeneous of the team via their work & profiles. 2) 
Selection to the sub-groups. 3) Group task analysis based on individual’s expertise: Time & Task management. 


Technological tools: Web GroupBlog VS Individual 


2. Sharing empathy: 

Team have a conversation with the end-users for collecting even richer and more provocative ideas. Team 
with different thoughts will take away the mindset, observe the world to find what end-users’ problems and how 
they are thinking, what they desire to do, and what they wish for. They collect insights by looking at the major 
problems as an insider, as if they were the users. 


Technological tools: Research Template 


3. Generate Idea: 

Members of the team virtually propose their ideas via online canvas based on information received from 
the users’ empathy stage. Members act as a team researcher and that resulting to different data from accessing end - 
users’ opinions. A structured brainstorming process. Taking one provocative question at a time via an online canvas, 
coding by color. Members freely generate ideas via an online Post-It note which allow for visual image and other 
media to be linked. Eventually, members categorize those ideas in the same shade of color, and naturally the ideas 
are ranked. This lets the group move into a process of grouping and sorting ideas. 


Technological tools: E-note Post-IT 
4. Competing prototype/solutions: 

At the core of the implementation process is prototyping: turning ideas into actual products and services 
that are then tested, evaluated, iterated, and refined. A prototype, or even a rough mock-up helps to gather feedback 
and improve the idea. Prototypes can speed up the process of innovation because they allow quick identification of 


strengths and weaknesses of proposed solutions and can prompt new ideas. 


5. Pilot-testing: will take their roles in filtering the ideas, an iteratively improve through the implementation for its 
ultimate solutions. 


Technological tools: E-research (form) 
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Table 1. Design Thinking process with a multidisciplinary Team 
Brown (2009) Inspiration Ideation Implementation 
Stickdorn& 
Schneider (2010) Exploration Creation Reflection Implementation 
Standford d.school 
(2010) Empathy Define /Ideate Prototype Test 
Coakley, Roberto, eae : 7 
& Segovis (2014) Inspiration Tdeation Implementation 
Glen, Suciu, Problem Finding, Visualization and Sense 
Baughn and Anson Observation Making, Ideation Prototype and Test Viability Testing 
(2015) 
Gachago, Morkel, ‘ Reflection and Resilience, 
Hitge, van Zyl and ean and ek Exploration and play Focus on practice, Change 
Ivala (2017) ty Agents 
Kirkman et al. (2002) 
confidence and trust 
-common goals 
-member roles 
- roles 
- relationship 
- virtual team 
performance 
- member training 
-evaluation 
Powell et al. (2004) 
-cultural difference 
expertise 
-relationship skills 
-choesion 
~communication - 
coordination 


-task- technology 
structure fit 


Shaw (2012) 
rewards routines 
resources 
- purpose 
- cleat roles 
- relationship skills 
Zimmer, Christina. 
(2015) 
-leadership 
- working according 
to the specified 
structure 
-communication 
- community 
Ghiringhelli & 
Lazazzara 
(2016) 
- training 
- locations 


- team selection 
- Positioning 

- Set team goals 
-motivation 

- team leader 

- member team 

performance 


- meeting structure 

- performance evaluation 
- electronic 
communication 

- working with 
technology 

- facilitating the team 

- Security 


172 


Synthesized tools 
of virtual team 
Hu (2015) 


-Innovation Flowchart 
- Google ventures design 
sprint 


-WeTHINQ 
- Question Ladder 


rocess in a Team mediated by communication technolo 


- Design thinking tool 
kit 


-WeTHINQ 


YouTube, Yammer, 
Wiki, Google Apps: 
Sites, Docs, Video, 
Maps, Calendar, 


- Design thinking mix 
tapes 
- IDEO design kit 
-Typeform -Smaply, -Userforge, - Boords, Mockingbird, UserTesting, HotJar, 
-Zoom MakeMy POP PingPong 
-Creatlr Persona, 
-SessionLab, 
-Stormboard, IdeaFlip 
-Join.me -Boards - InVision Studio - Ethnio 
- Free hand - Craft - Lookback 
- Coggle 
- Focus Booster 
- Smaply - Visual collaborative - Mockplus - Silveback 
- Usability tools work platform: Morally - Pop app 


Wiki, Google sites 


- Mind map tools: 
Mind Manager, Coggle 


Yammer, Elluminate 
Live (Blackboard 
Collaborate), Skyp, 
Google talk, chat 


Google sites, YouTube 


Blackboard, Google 
Apps 


gadget World Clock, 
g-mail 
Chastain & Nathan- | Sharing calenders, Blog, Application sharing - Web conferenceing, Streaming Audio/video, 
Roberts (2016) Web-based training, Video & Audio Narrated slideshow 
e-mail, datbases conferenceing 
Aritz, Waler, g-mail, messenger Google Docs, Dropbox phone calls, Facebook, LMS, Google Docs 
Cardon, & Li instant messaging, Skyp, 
(2017) Google Hangouts, 
Coference a call 
Larson, Leung, & Slack Dropbox, Google Face Time, Google LMS, Google 
Mullane. (2017) drive/Docs, Hangouts, Skype, Slack, drive/Does, Slack 
Teleconference 
application, WhatsApp, 
Google Hangouts, 
Edwards & Wilson e-mail Electronic display tools, 
(2017) Electronic meeting 
systems, Video & Audio 
onferenceing 
Mukherjee & Blog, Yammer Facebook, Google Disscusion Boards, Google drive/Docs, 
Natrajan (2017) drive/Docs, Google Hangouts, 
Yammer 
Ritika Tiwari Asana, Basecamp Trello, Slack Trello, Slack 
(2018) 


Orta-Castafion et al. 
(2018) 


Dropbox, Onedrive, 
Google Drive/Docs, 
Facebook 


Skype, Webex, 
Snapchat, Chat, Google 
Hangouts, Facebook live 


LMS, Google 
Drive/Docs 
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RQ 2. How the multidisciplinary team performed in the process of Design Thinking? 

The observation and interview are based on the framework of the Process, (People) Team, and Technology 
in the Design Thinking for the mini-courseware of the Multidisciplinary team which are explored in the dimensions 
of Context, Team structure, in the process of Design Thinking (Kirkman et al., 2002; Powell et al., 2004; Shaw, 
2012; Zimmer, 2015; Ghiringhelli & Lazazzara, 2016) 

Context & Team Structure: Twenty three participants, Master degree program from different 
background; Psychology, Information Technology, Instructors, Business, Computer scientist, Trainer, and 
Marketing. They enrolled in the Class of 2019, Educational Technology. The objectives of the class are to design a 
mini-courseware incorporating to a Learning Center to serve various kinds of targeted learners such as Primary 
school students, Undergraduate students, and Life-Long Learners. In the orientation session, the Design Thinking 
process was explained and Communication tools were suggested for members to review and agree upon. 

Communication Technology: Toolkits were introduced. Main features are Web Blog of the project 
consisting of: Individual and sub-group profiles; Project Management, “Persona” Template, Online Sticky Note, 
Whiteboard (JamBoard), Survey tools. 


Table 3 Members, design task in the Virtual Multidisciplinary Team 


Sessions Total of 21 Disciplines 
members 


Identy: what’s a 3 Psychology, Information Technology, Instructor 
matter! 

Open thinking for 3 Teacher, Sciencetist, Business 

creativity 


Digital Teachers Teacher, Computer Science, Management 


a Co-learning space 


Process 

Diverse inspiration: participants put their profiles on the web projects and expressed their inspirations and 
their opinions on the mini-courseware design project. Later, members decided to join the sub-group themselves into 
7 groups in a total of 21. Two could not decide to be members of any teams due to their feelings of lacking a proper 
knowledge to join the project. 


Team structure: The leaders form themselves for each sub-group and the whole project. The timeline and 
task was proposed and leave a week for members to agree, add opinions, or adjusted. 


Communication: Communication among themselves were not often, unless the deadline was approaching. 


Sharing Empathy: Members were trained and provided resources for the end-users interview. They 
worked as a group, making a decision on the questions to explore. Then, they worked in a pair and switch, using 
video conference to interview with the end-users. Individually, the team member wrote the data onto the template. 
As a group, they interpreted the data together and share with other groups. 


Team structure: Members whose expertises are not in the area of Information Technology and Computer 


Science/Engineering take the lead with a support from members from Social Sciences interpreting and synthesize 
the insights of users’ feelings leading to solutions or opportunities for change. 
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Communication: Sharing online Template “persona”; individuals in the team wrote the data very similar. 
Few data were slightly different from others. 


Generate idea: A structured brainstorming process occurred over an asynchronous mode of 
communication. Taking one provocative question at a time via an online canvas, coding by color. Members freely 
generate ideas via an online Post-It note which allow for visual image and other media to be linked. Eventually, 
members categorize those ideas in the same shade of color, and naturally the ideas are ranked. This lets the group 
move into a process of grouping and sorting ideas. 


Members put their own idea on the sub-group and share to the other group members. Using an online 
sticky note, others came to comments, clarify and adding more resources. 


Team structure: Members voluntary expressed and disseminate their ideas; while responded to other 
comments. 


Compete Prototyping/ solutions: A blog of team project displayed 5 learning activities and 2 wireframes; 
open 


Team structure: As a team, members in a sub-group responded to the comments and suggestions of other 
teams. Meanwhile, working as a commentator, individual member accessed other group works and contributed their 
opinions. 


Communication: There are some problems when an individual had to perfrom two functions as an evaluator 
and designer. Not all of them comments others, either responded to other members. The rating was offered by 
members from other groups. 


Pilot/testing: The E-questionaires were distributed to the potential end-users to gain their opinions, 
including a couple follow-up interview. Some notices on the results were put to the comments on the blog for 
further improved. 


Team structure: Members as a team of multidisciplinary acts as a research team; coordinate the results. 


Communication: Face-to-face is required from the members to clarify the whole process before further 
improvement. 


Discussions 


1. Diverse Inspiration: The multidisciplinary team was formed, acquired by a specific cognition and social- 
emotion in a sub group. 

1.1 Team confidence and trust must be earlier established with the team. The members selection to the 
team will help members to matched themselves and show their interpersonal skills to be taken to the sub-group. 
Members represented their expertise and eventually be selected from the subgroup. The selection of virtual team 
members is the selection to fit both expertise to match the task and interpersonal skills to be in the selected group. 
Interpersonal skills especially cultural awareness are more important as team members attempt to communicate 
effectively without relying on traditional non-verbal cues (Kirkman, 2002). 

1.2 Task & time management: Member recognition as unique expertise and contribution to the group will 
have consistently appeared on the group task & timeline. This stage is adapted to be a loose structure of the virtual 
multidisciplinary team that should be distributed and not strictly hierarchical structuring. Task and timeline, 
however, will support communication in the sub-group/community, leading and structure the work to be structured 
and followed by the group (Kirkman et al., 2002; Powell et al., 2004; Shaw, 2012; Zimmer, 2015; Ghiringhelli & 
Lazazzara, 2016) Team leader is still needed nominated by the members who for see the challenges of the lead the 
activity over the non face-to-face with his/her special characteristics. The discussion on the team leader who must 
have both task-related and socio-emotional strength to accommodate the cultural, experiential, and distance 
differences of virtual teams (Ford, Piccolo, & Ford (2017). 

1.3 Communication: Towards the communication in the context of the virtual multi-disciplinary team; 
Technology as a "task & time management" features support the task structuring, direction, feedback; as appeared 
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in the Bal and Gundry model (Ghiringhelli & Lazazzara, 2016) identified the areas in relation to team members such 
as team objectives for group motivation, meeting behavior derived from the trust, reward structure claimed by 
motivation and goal setting, and team selection occurred by supporting culture. In relation to technology, 
appropriate technology is selected to facilitate team effectiveness through a compelling communication channel 
(Ford, Piccolo, & Ford, 2017; Serrat, O., 2017; Alsharo, M., Gregg, D., & Ramirez, R. (2017) 

In addition, group communication via a group "task & timeline management tool" instantly notice the 
updated task of the group. The rapid responses by the system will stimulate team members for the reliable group 
performance. While personal and sub-group communication consistently follow-through will established the “trust” 
and “motivate” and bring up social existence that caused an awareness of the socio-emotion in the group (Kirkman 
et al., 2002) 


2. Sharing Empathy: This process of the group to experience the real users' problems. 

Originally, at the inspiration phase, designers look for the opportunities with the current status of users’ 
needs by observing and collecting insightful data from the users (Brown, 2019). At the stage of diverse group 
inspiration, the focus is shifted from the end-users to the multidisciplinary team. The purpose is to draw the pictures 
of users’ problems from different angles of expertise of the team. The team represents the end-users’, extreme 
points of view of problems and turns to a provocative ideas. An awareness of the diversity of the group, and 
welcome and persuade the group to the ultimate outcome. A clear and precise goal is announced on the homepage 
of the team. The group task is analyzed and along with the sub-team responsibility and timeline must be a group 
communication. 

2.1 Persona: Using the "persona" technique (Brown, 2009), members from their perspectives will have 
experience in an insightful of the end-users. Empathy brings in the centerpiece of a human-centered design process 
from multiple perspectives in intrepreting how end-users perceive the problems (Standford, 2012). Similar to a 
qualitative research approach that is a scientifically gather non-numerical data. The "Sharing Empathy” discloses 
why and how a certain phenomenon occur and can be employed across all disciplined. This method will help 
understanding end-users needs, through an observation leading to empathy, insights and analysis, gaining an 
unfiltered understanding of the user's experience, abilities, and constraints. (Glen, Suciu, Baughn and Anson, 2015). 
Eventually, the “Sharing Analysis” will be analyzed breaking apart the root causes of a problem from multiple 
perspectives. 

2.2 Problem oriented: The Communication Tool that supports analysis of problems and causes, cause-effect 
online collaborative diagram facilitate generative brainstorming from different points of view of the interdisciplinary 
group, rather than quickly jump into solutions. This problem orientation activity supports team to timely explore the 
problem from many different viewpoints, thereby remaining in the problem space for longer (Lawson, 2005 cited in 
Gachago, Morkel, Hitge, van Zyl and Ivala, 2017) 


3. Idea generation; 

At this stage, idea generation of the multidisciplinary team was brought into attention. This stage coincided 
with the work of Brown (2008) who proposed a divergent thinking from multidisciplinary; per se members from 
psychology, business/marketing, engineer. He postulated the “T-shaped” member; the vertical line-an individual 
with a depth skill, and the top of the “T” comes from openness, curiosity, and optimism from other people and 
disciplines. Technology supports idea generation and convergence easily present with color coding, simply as an 
online sticky notes, visual aids. (Brown, 2009). The colorful sticky notes could be easily rearranged, converging the 
idea, and move to other walls (online) (Ford, Piccolo, & Ford (2017). Visualization will help sense-making for 
information become meaningful. Visual aid serves as a tool in grouping pieces together a myriad of unorganized 
explicit artifacts in a memory for logically compare and contrast. It will spark a mental connections through a 
various perspectives of members (Glen, Suciu, Baughn and Anson, 2015). One best practice similar to the “Sharing 
space” that displays ideas to be mended together. An example of “InnoCentive” website, non-profit and business 
company involvement, allows currently more than two-hundred-thousands of members with multi-disciplines 
around the world to challenge and reward effective design solutions. 


4. Competing Prototyping 

Seven design outputs from the process of thinking from multi-disciplinary team found to be provocative as 
the thinking had been through the divergent and convergent thinking. Prototype is a step to make a convergent idea 
to be concrete. It is an iterative generation of artifacts intended to answer questions that getting closer to the final 
product/solution (Stanford, 2012). Competing Prototype elicit useful opinions from team to be refined and 
improved. 
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5. Pilot Testing 

This stage could be a pre-execution to the vision and trial by end-users. The pilot/testing mode is beneficial 
for feedback about the prototype. It is literally opportunity to gain empathy for targeted end-users. An electronic 
assessment system could embarrassed a direct report from all perspectives of users and other teams (Kirkman, 

2002). At this point, the design could iteratively to the prototyping or either bounced back to the Sharing Empathy 
process, acquired insightful of the end-users and redefined the problems, and work again through the ideate cycle. If 
the prototype could go far further than the testing of idea but interaction with users, elicit questions should remain 
“how” and “why” end-users have trouble with. The testing requires more than technical rationality of the 
courseware. 

It could also a group reflection in action with other professions (Glen, Suciu, Baughn & Anson, 2015). The 
testing process is a reflective practice and should be continuous to promote a professional skills and resilience 
(Gachago,et.al.,2017; Lawson, 2015). This pilot testing could help minimizing the so-called by Taheri et.al. (2016) 
“creative overconfidence” caused by a lack of competence skills, overemphasis on technologies and tools rather than 
practices. Roles of team leaders facilitate members to genuinely and positively contribute towards the success of the 
team. Brown (2008) mentioned the “Design Thinking” is never ending cycle thought, thinkers (Multi-disciplinary) 
will never stop generating and competing idea. 
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Abstract 


The purpose of this study was to create the digital learning by Digital Learning Analytics’ method as 
the appropriated tool to develop the creative thinking in Thai students. The study consists of five procedures 
which are, first, studying the documents, principles, theories and the researches which related to the digital 
learning, learning analytics and creative thinking. Second, designing the digital learning combine with the 
learning analytics to develop creative thinking. Third, creating and developing Digital Learning Analytics. 
Fourth, applying Digital Learning Analytics to printing product course for education technology. Fifth, 
evaluating the using of Digital Learning Analytics and Creative Thinking test. The sampling group of this study 
is 30 Thai students, who have basic technology skills. The result of this purposive sampling was analysed by 
these following methods, Mean Method and Satisfactory Evaluation. Finally, I present the using of Digital 
Learning Analytics, which consisted of 7 steps 1. Import Data 2. Identify problem 3. Plan the concept 4. Create 
a prototype 5. Utilize by user 6. Assessment process 7. Modify the route. Those results will be evaluated by the 
experts in Digital Learning Analytics and also the creative thinking Thai students will be evaluated as well. 


Introduction 

Nowadays, the world society is stepping into the digital society in which people communicate via the 
network, access information with just a click of a finger. Technology and computers have been developed and 
used in both the business world and personal use or for other purposes as quickly and widely as needed. 
Resulting in the amount of information that newly created in each day. These data are a valuable knowledge 
resource if we can use it. (Yaowaluk and Sophon, 2561) Information and activities through the online world is 
increasing every year. Data from Thailand Internet User Profile 2018 from the Electronic Transactions 
Development Agency (ETDA) found that Thailand has more than 57 million internet users, for social media 
regularly reaches 51 million. About 55.56 million people are mobile phone users, and about 46 million social 
media users use its through smartphones. Thai people use the internet for nearly 10 hours a day on 
workdays/school days and 11 hours on holidays. It is also found that the behavior of Thais who use the internet 
via smart phones include talking 77%, watching videos 75%, playing games 66%, searching routes 64% and 
making financial transactions 56% (Rohan, 2018). This dramatic increase in usage affects thinking, life, and 
information understanding bring data together from various media and the ability to analyse the relationships of 
that data, enable us to improve business operations, sales and marketing, able to take advantage of competitors 
or create business opportunities. As can be seen from the data collected to lead to the analysis of solutions. For 
example, from the article “Deciphering the unstoppable Netflix and the role of Big Data” analyze that Netflix 
has achieved rapid success by collecting audience data in terms of viewing history, pressing stop playback, used 
devices, search, rating, etc. to create Big Data which analyzed from past movies selection that which movies 
viewers would like to watch next. Then presenting more directly or closely to the preferences and tastes of 
consumers, affect the experience and satisfaction. It results in more than 137 million subscriptions (data at the 
end of 2018). These analyzed data is not only used to increase business opportunities but also educational has 
uses data collection for analysis in learning. 

Learning analytics is the process of measuring, collecting, analyzing and reporting learner results for 
the purpose of understanding the learners. Both problems and factors that promote learners learn in order to 
manage the learning environment to maximize results. (Jaitip, 2016) The software features of the data analysis 
software set include Content analysis, Discourse analytics, Social learning analytics, Disposition analytics by 
analyzing various data from the collected database, it is an important part of the operation. If there is enough 
data and covering relevant factors, it will help the result of the analysis to be precise and accurate. Study 
analysis procedure begins with bringing the information that needs to be done in a form that is ready to be 
processed by using technology or the set of instructions and the model created in order to use the information 
that has been analyzed and translated or interpreted according to the context and the learning environment. It is a 
support for solving problems and increasing the efficiency of digital learners. 
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Learning in the digital age is a process or method that a person uses to create meaning from 
information, environment, social stimuli that are received sensory, resulting in knowledge, skills and attitudes 
when systematically developed, it can be predicted to be effective at full capacity (Jaitip, 2016) that is suitable 
for the era. Presented on the platform to reach more young people with the internet and portable communication 
devices like smartphones being an important factor in the digital age. People are able to use technology 
aggressively and perform a variety of activities within their fast communicating all the time. There are many 
ways to access information, sharing, exchanging ideas, freedom to express ideas and express themselves through 
social media and also self-learning at anytime, anywhere through resources on a vast network. Learning 
activities has changed not only in the classroom but also an access the classroom freely in anytime and 
anywhere which using digital technology as a creator (Sungkawadee & Keawurai, 2017). Resulting in methods 
of accessing students' knowledge to use technology as a tool and develop oneself to create knowledges and skills 
that are continuous and sustainable. Digital tools for learning mean software and platforms for teaching and 
learning as well as text, images, audio, video, and programs for editing the digital content, working together and 
sharing resources with others to communicate knowledge (Interactive Teaching in Languages with Technology, 
2017) 

Creativity is the ability of the brain to think far and wide. Many aspects of its create new ideas which 
are differ from before. It is the ability to see the various relationships around, learn to understand until the 
reaction occur to the imaginative thought, which is an important aspect of creativity, leads to the creation of new 
things to solve problems which will require integration from all experience and knowledge (Charnnarong, 2003) 
By educational in Thailand at present, students are encouraged to create creative ideas that rely on technology 
and the internet to apply their competencies in accordance with Thailand 4.0 policy. Thailand educational 
institutions must integrate creative development with the methods and learning activities of the digital age that 
can access information in anywhere, anytime, as well as support lifelong learning in various courses therefore, 
focuses on the students to use creative thinking processes to solve learning problems 

Learning in printing production courses focuses on students to be able to produce publications. Not only 
understand the principles of production but the design of publications is also an important part. The design of print 
media requires principles of font or text design, material, illustration, free space and other components for the 
publication to meet the objectives, target group and communicate effectively 


2. Research Objectives 
The objectives of this study was to create the digital learning by Digital Learning Analytics’ method as 
the appropriated tool to develop the creative thinking in Thai students. 


3. Sample size 30 undergraduate students in Educational Technology department 


4. Research Methodology 
4.1 Studying the documents, principles, theories and the researches which related to the digital 
learning, learning analytics and creative thinking. 


4.2 designing the digital learning combine with the learning analytics to develop creative thinking by 

divided into the following steps 

1. Defining the conceptual framework from the data obtained from the analysis in step | and create it 
in the format 

2. Designing content and learning activities in the printing production course by using the creative 
development process, the SCAMPER model, consists of Substitute, Combine, Adapt, Magnify, or 
Minimize or Modify, Put to Other Uses, Eliminate, Reverse or Rearrange (Sirichai, 2015) 

3. Designing Digital Learning in learning. It can be divided according to usage of the teaching and 
learning activities into 5 categories (Hart, 2017; Poore, 2013), including 1) Teaching management 
tools 2) Content development tools 3) Website resource tools 4) Social tools and 5) Personal tools 
and job development (Kobsuk, 2018) 

4. Designing evaluation form by using the Reisman Diagnostic Creativity Assessment (RDCA) 
which is a test that is built on from the Torrance Tests of Creative Thinking (TTCT), the most 
popular test for measuring creativity that was developed in the Thai version (Naruertep & Charun, 
2018) RDCA can measure 11 elements of creativity including originality, fluency, flexibility, 
thoughtful thinking, vague solutions, resistant of premature closure, independent thinking, 
convergent thinking, risk taking, motivation, and external motivation. 

5. Designing analytics for learning analytics by selecting the type of discourse analytics. This type of 
analysis is to store meaningful information in the communication interaction of learners with 
Digital Learning tools in order to correct and improve efficiency of learners' creativity 
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4.3 Creating and developing Digital Learning Analytics. Develop tools for online teaching and 
learning via the internet (LMS) with Moodle software, which is a software for creating web-based tutorials. By 


determining to have a lesson management system which supports 3 user groups that are administrators, teachers, 


and learners. Provide an effective web-based learning environment. This system was developed based on 
Software Open source. Then create lessons in the Printing Production for education and communication course. 


According to the following learning units 


4.3.1. Printing History 


1.1 European print production technology 
1.2 Asian publishing production technology 


4.3.2. Educational publications and principles of educational publications design 
4.3.3. Printing systems and materials related to printing 

4.3.4. Applying graphic design principles to print design 
4.3.5. Newspaper publications 
4.3.6. Magazine and periodical publications 
4.3.7. Book publications 
4.3.8. Specialized publications (brochures, small books, booklets, flyers, posters-newsletters) 
4.3.9. Specialized publications (flyers, posters-newsletters and other specialized publications) 
4.3.10. Principles of designing original publications 


10.1. Publishing publication 
10.2. Principles and concepts of electronic publications design 
4.3.11. Electronic publications design 


4.4 Applying Digital Learning Analytics to printing product course for education technology. 


4.5 Evaluating the using of Digital Learning Analytics and Creative Thinking test.Assess satisfaction and 


assess the creative ability of learners before-after using Learning Management System Implementation (LMS) 


Import Identify cs 
Data problem i 
concept 


Create 
a 
prototype 


Utilize 
by user 


Assessmen 
t process 


Analytic & Develop 


| Modify 
the route 


Model of Learning Analytics of Digital Learning in Printing Production course for 
Education Technology to increase Creative thinking in Thailand 


Table: 1. Model of Learning Analytics of Digital Learning in Printing Production course for Education 
Technology to increase Creative thinking in Thailand 


Model Detail Digital Tools Creative Thinking Productive 
1. Import Data | Data Import Quizletstudy Online pre-test 
- It is the import of data. | Google Form SCAMPER creative 
by testing the creativity development 
of students. process 
2. Identify Problem Analysis Giving Opinion Substitute Discussion of 
problem - From the test scores, it | Reflecting Combine messages or 
was found that the Knowledge and Adapt ideas for 


creative scores in 
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Model Detail Digital Tools Creative Thinking Productive 
is still Creating links in Magnify or working in the 
missing, so promotion the group area Minimize or Modify | classroom 
must be carried out Put to Other Uses 
3. Plan the Action Planning Create mind maps | Eliminate Mind map 
concept - Use the analyzed by using the Reverse or creation Work 
problems to design the online graphic Rearrange outline 
problem solving plan by | map tool. 
brainstorming 
- Using the creative 
promotion process 
4. Create a Create templates / Create | - Online Learning 
prototype media / Create online document sharing Management 
activities to promote tool System 
creative - Tools that help 
to display the idea 
of a work that can 
use text, images, 
video, audio 
- Learning 
Management 
System (LMS) 
tools 
- Classroom 
website 
5. Utilize by Bring the created LMS Printing 
user template to be used by Google classroom Product 
relevant parties such as - Name card 
students and teachers. - Brochures 
- Cut out 
- Newspaper 


6. Assessment 
process 


Evaluation 

- Take post-test to 
measure creativity 

- Inspect workpieces 
from online activities to 
check creativity during 
school 

- Analyze statistics of 
each type of media 
access in order to 
examine access 
behavior, frequency 
level of media access 
that affect creativity 


Assess in LMS 
Statistical 
Analysis Website 


7. Modify the 
route 


- Bring the results from 
the evaluation in all 3 
dimensions to analyze 
and find solutions for the 
highest learning 
efficiency 

- Making factual 
solutions 

- Go back to the template 
drafting process to 
improve 


- Online 
document sharing 
tool 

- Tools that help 
to display the idea 
of a work that can 
use text, images, 
video, audio 

- Learning 
Management 
System (LMS) 
tools 

- Classroom 
website 


Online post- 
test 


Statistical 
analysis from 
attendance and 
activities 
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5. Analysis and Discussion 

Assessing the suitability of the formats Learning Analytics of Digital Learning in Printing Production 
course for Education Technology to increase Creative thinking in Thailand by applying the evaluation form to 5 
educational technology experts 


Table 1. Results of data analysis of the suitability of the overall model 


Experts Level of opinion 

Evaluation List = SD. High 
History of model development 4.40 0.70 High 

Process of format 4.50 0.53 High 
Results of the format 4.40 0.70 High 
Functional Use 4.50 0.53 High 
Reflection of the format 4.50 0.53 High 
Total Average 4.46 0.60 High 


From Table 1, the results of the data analysis on the suitability of the formats Learning Analytics of 
Digital Learning in Printing Production course for Education Technology to increase Creative thinking in Thailand 
found that the evaluation results were at a high level in all items. The process of the format, the results of the 
format and the suitability of the implementation of the format have the same mean score and standard deviation 
which is 4.50 and 0.53 respectively. 

The results of the comparison of the average score of the creative thinking before and after learning of 
the experimental group of 30 students 


Table 2. The results of the comparison of the average score of the creative thinking 
scores before and after learning of the experimental group. 


_ Level of Creativity number (%) 
Components of Creativity 


Before after 
Originality 66.25 89.34 
Flexibility 45.56 78.43 
Thoughtful Thinking 50.14 82.27 
Vague Solutions 33.64 80.65 
Resistant of Premature Closure 64.33 75.89 
Independent Thinking 56.09 81.23 
Convergent Thinking 45.05 88.19 
Risk Taking, 34.78 86.92 
Motivation 48.63 79.54 
External Motivation 53.25 70.08 


From Table 2, the comparison of the average scores of the pre-test and post-test of the experimental 
group showed that there was an increase in all components of creativity. The pre-test, Vague Solutions has the 
smallest mean score and Originality has the highest mean score. After experiment, the researcher found that 
Originality has the highest mean, and Resistant of Premature Closure has the smallest mean. 

Analysis of the interaction of students’ interaction with Digital Learning tools in learning activities 
through Learning Management System (LMS) 


Table 3. Analyzing the interaction of learners' communication with Digital Learning tools 


Digital Learning tools Average time spent (%) Number of students using (“%) 
PDF file 15 5, 
Video Instruction 45 35 
Social Media 40 60 
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From Table 3, it is found that the analysis of the interaction of students' communication with Digital 
Learning tools, The tools that students using the most is Social Media, followed by Video Instruction and PDF 
file. Moreover, spending time in learning with Digital Learning tools found that students spend the most time on 
Social Media, followed by Video Instruction and PDF file. 


6. Discussion and Conclusion 

From the study, development in the format of Learning Analytics of Digital Learning in Printing 
Production course for Education Technology to increase Creative thinking in Thailand consisted of 7 steps 1. 
Import Data 2. Identify problem 3. Plan the concept 4. Create a prototype 5. Utilize by user 6. Assessment process 
7. Modify the route, when using that format for data analysis from the suitability evaluation form of the format 
that has been evaluated by experts. It found that the suitability of the overall format has a high level. The average 
value is 4.46 and the standard deviation is 0.60 which shows that this format is suitable for applying the activities 
of the Printing Production course. 

In addition, students have a higher creative average score by using Digital Learning Tools, which found 
that social media has the highest number of uses and duration It corresponds to the concept of using social media 
in teaching and learning. It is an important issue at present. Teachers can apply this information to stimulate 
students' interest and use it as a technique to help them achieve academic achievement (Kobwit, 2011) 

Moreover, teachers must have the knowledge and capability to use these tools in factual and sustainable 
teaching and learning processes. The teacher understands the techniques/tactics of the tools combined with 
teaching strategies and creating an interesting format for that student, is very challenging for teachers (Office of 
Technology for Learning and Teaching, 2011), which is consistent with the results of the above research that 
provides empirical evidence from data collection. As well as the time period for using it with the online learning 
management system, then analyzed for learning (Learning Analytics) is a tool to help learning design to meet the 
objectives, respond and support learners for effective learning. 
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Abstract 


This study determined teacher candidate students’ perception of online course quality and factors impacting 
on their course satisfaction and perceived learning. The results showed that facilitation and instructional strategies 
were significant factors, and there were positive relationships between self-efficacy and students’ course satisfaction 
and their perceived learning. The findings indicated that facilitation is an urgent area of the online course that needs 
to improve to enhance the quality of online education. Our research also revealed that organization, facilitation, and 
instructional strategies significantly affected student course satisfaction while facilitation and instructional strategies 
significantly impacted their perceived learning. Regarding online learning self-efficacy used as a predictor for 
students’ course satisfaction and their perceived learning, our findings showed that confidence level positively 
influenced student course satisfaction and perceived learning. 


Introduction 


In the U.S. higher education, 29.7% of all students are taking at least one distance course. The total distance 
enrollments are composed of 14.3% of students (2,902,756) taking exclusively online courses and 15.4% 
(3,119,349) who are taking both distance and non-distance courses (Allen & Seaman, 2017). With the demands of 
online learning, almost 40% of administrators plan to increase their budgets in the next year, and 73% of schools 
decide to offer online programs based on the potential student enrollment (BestColleges, 2019; Venable, 2018). 
However, the rapid growth of online learning in higher education requires quality control issues (Andrade, 2015; 
Venable, 2018). Thus, numerous research findings pointed out areas for future investigations related to the quality of 
online education including the nature of course designs and design aspects (Boling, Hough, Krinsky, Saleem, & 
Stevens, 2012; Holzweiss, Joyner, Fuller, Henderson, & Young, 2014; Kuo, Walker, Belland, & Schroder, 2013; 
You, Hochberg, Ballard, Xiao, & Walters, 2014); online community and student engagement and instructional 
techniques (Holzweiss et al., 2014); technologies use, faculty support (i.e., knowledge, skills, and pedagogical 
strategies) (Boling et al., 2012); online scaffolding, quality of interactions, and instructor feedback (Rourke & 
Coleman, 2010). There is a lack of researches conducting the teacher candidate students regarding their perception 
of the quality of online courses. 

Recently, the College of Education, Texas Tech University developed the Quality of Online Education 
(QOE) framework to guide instructors to develop quality online courses based on current online course standards 
and guidelines. The framework focuses on the interaction among instructors, students, and contents, and it includes 
six core actions between the agencies to provide meaningful learning experiences. 


Action 1 - Organization: refers to an organization of the course structure and information presenting through the 
course (e.g., syllabus, schedule, modules, assessment, etc.) to ensure that the course is presented in a consistent way 
in terms of aligning course objectives, learning activities, and assessment and is to minimize barriers to students 
understanding and confusion. 

Action 2 - Instructional strategies: refers to all teaching approaches and methods that an instructor may take to 
engage students in the learning process (e.g., activities, assignments, and assessments). to foster student learning and 
aid them in mastering their learning process. 
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Action 3 - Assessment: refers to designing and creating types of assessments by using a variety of methods to assess 
student performance during the learning process to ensure that they achieve learning objectives and meet teaching 
expectations. 

Action 4 - Materials: refers to providing accurate, current, and relevant contents using a variety of media (e.g., 
PowerPoint, Videos, online articles) that effectively delivers important knowledge and skills and visually appeals to 
students. 

Action 5 - Facilitation: refers to actively monitoring students’ learning progress and scaffolding their learning using 
a variety of available resources and technology. 

Action 6 - Interaction: refers to effective communications and collaboration between an instructor and students by 
using a variety of available resources and technology to construct knowledge, provide constructive feedback, and 
build a sense of community that appreciates multiple perspectives and supports each other’s learning. 


The College of Education’s QOE framework represents the interrelated actions among three main agencies 
(i.e., faculty, students, and content) in online education environments to facilitate the faculty’s implementation of 
quality online courses. The framework focuses on interrelations among three agencies (i.e., faculty, students, and 
content) that play important roles in online education. It also provides a systematic approach to the interdependent 
nature of online environments with a graphical representation (Agency — Action - Component [What-is] - Method 
[How-to]). The framework provides various practice examples for the college’s instructors, and it emphasizes the 
importance of context (e.g., discipline, students, setting, or system) that affects learning goals and activities in online 
courses. However, there is a lack of research on the differences of online course quality between instructors’ and 
students’ perceptions. Also, the relationships among the six actions have not been investigated yet. 


Purpose of the Study 


The study determined students’ perception of online course quality and factors impacting on their course 
satisfaction and perceived learning. It allowed us to identify online course quality discrepancies and improving areas 
and define factors affecting student learning and satisfaction as well as figure out define factors contribute to their 
outcomes. With the increasing numbers of students enrolled in online higher education settings, we saught 
meaningful implications on both practice and research in designing online courses through this study. Research 
questions include: 

1. Which areas of online courses need to be improved? 

2. What are predictors for student satisfaction and their perceived learning in an online course? 

3. What are the relationships between online learning self-efficacy and their course satisfaction and 

perceived learning? 
Methods 


A quantitative approach was used to collect students’ perception about their online courses to describe 
trends and compare groups using statistical analysis and interpret results to answer the research questions (Creswell 
& Creswell, 2018). 

Sample. 246 undergraduates, 25 to 35 years old at the COE, the Southwestern University voluntarily took a 
survey. The participant was taking a course “Application of Technology in Education” which is to help them develop 
instructional skills associated with the use of technology as an educational and assessment tool for instruction. 

Instruments. The online survey collected data from students’ perception and experience in their online 
courses. It includes demographic information, experience, course satisfaction, perceived learning, six actions (i.e., 
Organization, Facilitation, Interaction, Instructional Strategies, Materials, and Assessment), online learning self- 
efficacy, challenges, and suggestions. The survey items used Likert scale (from 1- Strongly Disagree to 5- Strongly 
Agree) related to online teaching-learning aspects and open-ended questions to gather their perception about their 
online course. It took about 15-20 minutes to complete the survey. The survey was created, distributed, and recorded 
responses by using the Qualtrics system. 

Data collection procedures and analysis. After obtaining permission from instructors and students, the 
survey link was distributed to participants through a course that they are taking. They completed the survey online, 
and their responses automatically recorded in the Qualtrics system. Next, the pseudonymization technique was 
performed to maintain students’ unidentified personal identifiers. All raw data was assigned to each response in a 
database using SPSS 25.0 for analysis. After cleaning the database to check for data entry errors, data was recorded 
and computed new variables (e.g., Facilitation, Organization). It was explored to inspect trends in the data and check 
for the normal distribution of all constructs. The correlation and multiple regression were used to identify the 
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relationship between six actions and online learning self-efficacy affecting students’ satisfaction and perceived 
learning. 


Results 
Regarding improving areas, the results revealed that the organization (M=4.17) was the highest ratio among 
the QOE six actions while facilitation was the lowest level (M=3.74) (See Table 1). This finding indicated that 


facilitation referred to an area to improve. 


Table 1. Mean Scores of Students’ Perception 


N Minimum Maximum Mean Std. Deviation 
Organization 264 2.50 5.00 4.1711 .60564 
Facilitation 264 1.17 5.00 3.7462 .79419 
Interaction 264 1.40 5.00 3.9205 .72071 
Instructional strategies 264 1.20 5.00 3.8803 83054 
Materials 264 1.00 5.00 3.9545 .76975 
Assessment 264 1.00 5.00 4.0019 .71246 


Multiple regression analysis was conducted to test if the six actions significantly predicted participants’ 
ratings of course satisfaction and perceived learning. For the course satisfaction, the results indicated that the set of 
independent variables explained 56.5% (p = .001 ) of the variance in the course satisfaction with three of six 
variables: organization (B = .234, p = .001), facilitation (B = .268, p = .001), and instructional strategies (B = .272, p 
= .004 ) having a significant influence on student course satisfaction (See Table 2). As a result, organization, 
facilitation, and instructional strategies had the greatest impact on course satisfaction. Interestingly, although 
interaction, materials, and assessment were predictors to the quality of online education (Yang & Durrington, 2010), 
they did not play essential roles in making students satisfied and in providing students with the authentic learning 
experience in their online course. 


Table 2. The Results of Multiple Regression Tests Regrading Course Satisfaction 


Coefficients’ 
Standardized 
Unstandardized Coefficients Coefficients t Sig. 
Model B Std. Error Beta 
1 (Constant) -.434 .284 -1.525 129 
Organization 366 109 234 3.345 001 
Facilitation 320 .099 .268 3.243 .001 
Interaction 073 .094 .056 775 439 
Instructional strategies 311 107 272 2.901 .004 
Materials -.003 107 -.002 -.027 979 
Assessment -.006 116 -.005 -.055 956 


Dependent Variable: Course satisfaction 

To perceived learning, the regression results revealed that the set of independent variables explained 62.1% 
(p < .001) of the variance in the perceived learning with two of six variables: facilitation (B = .225, p = .004 ) and 
instructional strategies (B = .250, p = .005) having a significant influence on student perceived learning (See Table 
3). Consequently, our findings highlighted that organization, facilitation, and instructional strategies significantly 
affected student course satisfaction while facilitation and instructional strategies significantly impacted their 
perceived learning. 
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Table 3. The Results of Multiple Regression Tests Regrading Perceived Learning 


Coefficients* 
Standardized 
Unstandardized Coefficients Coefficients t Sig. 
Model B Std. Error Beta 
1 (Constant) -.244 248 -.984 326 
Organization 157 .096 .108 1.647 101 
Facilitation 251 .086 225 2.918 .004 
Interaction 143 .082 117 1.738 .083 
Instructional strategies .267 .093 .250 2.852 .005 
Materials .012 093 .010 124 .902 


Assessment .197 101 158 1.954 .052 


Dependent Variable: Perceived Learning 


Regarding online learning self-efficacy used as a predictor for students’ course satisfaction and their 
perceived learning, correlations were conducted to test if it positively correlates to student satisfaction and perceived 
learning. The result showed that online learning self-efficacy and course satisfaction were positively correlated r 
=.632, p <.001, and they had a positive relationship with each other (See Table 4). 


Table 4. The Correlation Between Online Learning Self-Efficacy and Course Satisfaction 


Correlations 
Online Learning 
Self-Efficacy Course Satisfaction 

Online Learning Self-Efficacy Pearson Correlation 1 632" 

Sig. (2-tailed) 000 

N 258 258 
Course Satisfaction Pearson Correlation .632™* 1 

Sig. (2-tailed) 000 

N 258 264 


For the perceived learning, the result reported that online learning self-efficacy was positively correlated, r 
= .647, p <.001 with perceived learning, and they positively related with each other (See Table 5). Therefore, our 
findings revealed that confidence level positively influenced student course satisfaction and perceived learning 
(Alqurashi, 2017; Kuo et al., 2013; Xiao, 2012). 


Table 5. The Correlation Between Online Learning Self-Efficacy and Perceiving Learning 


Correlations 
Online Learning 
Self-Efficacy Perceived Learning 

Online Learning Self-Efficacy Pearson Correlation 1 647" 

Sig. (2-tailed) 000 

N 258 258 
Perceived Learning Pearson Correlation 647" 1 

Sig. (2-tailed) .000 

N 258 264 
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Discussion 


Overall, our findings indicate that facilitation is an area to improve. The findings also show that 
organization, facilitation, and instructional strategies had the greatest impact on course satisfaction while facilitation 
and instructional strategies significantly impacted their perceived learning. Online learning self-efficacy and course 
satisfaction have positively relationship with each other while it also positively relates with student perceived 
learning. 

Regarding course organization, it includes several aspects to ensure the quality of online courses involving 
course structure, course introduction, feedback, consistency in information delivery, relevancy, learning objectives, 
and technical support (Bickle & Carroll, 2003). Courses designed with facilitation increase students’ performance 
and satisfaction by promoting interactions and collaboration. Instructors should maintain the alignments of learning 
objectives with the instructional methods and learning activities with the assessment. The well-organized course will 
reduce students’ confusion. 

Facilitation plays an essential role in enhancing the online course, which confirmed from previous studies 
(Bigatel, Ragan, Kennan, & Redmond, 2012; Ragan, 2008). It also helps students become responsible for their self- 
assessing and learning progress and encourages them to solve problems (Eom & Ashill, 2016; Jaggars & Xu, 2016). 
To increase facilitation level of student learning in online courses, instructors should provide opportunities for 
students to reflect on course activities (e.g., reflection journal), provide specific feedback on their assignment and 
activities, regularly monitor their learning progress, encouraged them to ask questions, and actively participated in 
online discussion by replying to students, summarizing discussion, or asking questions to students. 

In terms of online learning self-efficacy, it plays a significant role in the performance of online learners 
(Taipjutorus, Hansen, & Brown, 2012). For example, students with a higher level of self-efficacy have better 
learning performance and increase their learning confidence which makes them satisfied with their learning 
(Taipjutorus et al., 2012). Moreover, students with higher self-efficacy are more likely to perform well and persist in 
online courses. The level of self-efficacy increases when they successfully complete their activities (Alqurashi, 
2017). To increase students’ online learning self-efficacy, instructors can build up student self-efficacy with 
supportive communication and constructive feedback to guide them through the tasks and activities and motivate 
them to through challenge projects to boost their best effort to achieve the learning goal. They are also positive and 
supportive to students to continue participating in the online course. 


Conclusion 


The study determined students’ perception of online course quality and factors impacting on their course 
satisfaction and perceived learning. The research findings showed that facilitation is an urgent area of the online 
course that needs to improve to enhance the quality of online education. The findings also revealed that 
organization, facilitation, and instructional strategies significantly affected student course satisfaction while 
facilitation and instructional strategies significantly impacted their perceived learning. Regarding online learning 
self-efficacy used as a predictor for students’ course satisfaction and their perceived learning, our findings revealed 
that confidence level positively influenced student course satisfaction and perceived learning. Therefore, it is 
important for instructors to design well-structured courses, maintain regular communication and presence in their 
courses, and promote student engagement. This can lead to greater student perceptions of learning and satisfaction. 
The practical implication of this study identified key components of high-quality online courses based on the 
instructors’ and students’ perceptions to improve the quality of online courses. We believe that this research 
generated useful knowledge on how to design and develop online courses based on the QOE framework (six actions) 
which provide students with the authentic learning experience and promote their learning satisfaction in the online 
environment. 
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Abstract 


One of the issues in the field of open and distance learning has been the dropout rate. However, due 
to the flexible nature of open and distance learning, a great deal of studies that focused on dropout rate was 
criticized. This paper intended to elaborate the dropout issue by using the Anadolu University’s distance 
education system as a case. One of the important points that this paper focuses on the concepts ‘active 
learner’ and “passive learner’ that usually do not seem in face-to-face settings. 


Introduction 


Open and distance learning is a structure based on self-learning and leaving the responsibility of 
learning to the individual to a large extent (Holmberg, 2005). Learners are responsible for their own learning 
experiences. They may encounter different challenges during the learning process and may have to drop 
out. Dropout is an important problem in terms of open and distance learning. 

In the literature, there are many definitions regarding dropout. The students, who prefer other 
educational institutions, leave their institutions or end their educations due to economic, social or family- 
related reasons while their educations are still continuing, are defined as dropout students (Chen, 2008). 
According to Cheung and Kan (2002), the students, who drop out, are defined as the individuals who 
officially leave the school, leave without informing the school or cannot continue to a department during a 
term. According to another definition, the dropout students are the ones who voluntarily leave the 
department after paying the tuition fee and completing add/drop procedure (Kaplan, Peck and Kaplan, 
1997). Although it is known that learners are willingly enrolled in open and distance learning, the dropout 
rate is higher compared to traditional education (Moore and Kearsley, 2005). When the literature is 
examined, the dropout rates in open and distance learning programs are between 25% and 40% (Lee and 
Choi, 2011). In this field, a great number of studies have been conducted on the reasons of dropout rather 
than identifying who are the dropout students. In the literature, a different definition for Dropout was made 
and “theoretical break” was used. This definition is based on the probability of students not continuing a 
specific academic programme following several consecutive semesters of “theoretical break’, and is highly 
adaptable to institutions offering distance education with no permanence requirements, that is ones offering 
the possibility of taking breaks (Grau-Valldosera & Minguillon, 2014). 

However, there are the concepts of 'active learner’ and 'passive learner’ which are used in open and 
distance education systems but not used for face-to-face education. In the institutions providing education 
through open education, it can be stated that the concepts of active and passive learner should be discussed 
together with the concept of dropout. 

Open education is used for the practices that provide large masses with the opportunity of internet- 
supported and relatively low-cost education (Aydin, 2011). In the broadest sense, open university refers to 
the elimination of all possible political and practical obstacles to reach learning resources. It is an 
instructional approach that does not have a time and space limitation, is based on self-learning of learners, 
is supported by open sources, and in which the support of instructors is obtained when needed (Distance 
Education Dictionary, 2019). In Turkey, Anadolu University is among the mega universities in terms of the 
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number of open education students. An important problem of distance education institutions is dropout. 
Examining the number of dropout students in open education faculties in Turkey in terms of dropout will 
shed light on the subject of active-passive learners of open universities. In this research, the status of active 
and passive learners receiving open education in Turkey was examined within the scope of dropout. 


Open Education Enrollment Types 


In this section, how students are admitted to open education faculties in the Turkish higher 
education system was examined before addressing the definition of the concepts of active learner and 
passive learner. According to the data of September 2019, there are totally 207 universities in Turkey. 129 
of them are state universities while 78 of them are foundation universities (Higher Education Information 
Management System, 2019a). Three of these universities incorporate open education faculties. In the higher 
education system, there are different types of enrollment for the candidates who want to enroll in open 
education faculties. These types can be listed as SSPS (Student Selection and Placement System) new 
student enrollment, vertical transfer, lateral transfer, second university and degree completion. 

The most known way to receive education in the higher education system in Turkey is the Central 
Placement Examination. Candidates are required to take transition to higher education examination 
conducted by the Center of Assessment Selection and Placement (CoASP). A similar situation is in effect 
for both face-to-face and open education. Student quotas of open education faculties are determined by the 
Council of Higher Education (CoHE). 

According to the examination conducted by CoASP, the candidates, who have graduated or will 
graduate from at least a high school or an equivalent school, need to take the Higher Education Institutions 
Examination in order to gain the right to new enrollment (CoASP, 2018). For the candidates taking the 
examination and receiving the required score for making a selection among the associate degree and 
undergraduate programs, the department and program lists of the open education faculties are published 
and selection procedures are followed. According to the results of the placement to be performed by CoASP 
within the quotas specified by CoHE, the right to new enrollment is gained. Following the placement, 
enrollment procedures are completed within the announced dates. Additional placement procedures are 
applied for the candidates who cannot be placed in any program in line with the available quotas. Similarly, 
the selection and placement results are announced, and the new enrollment process is completed. 

Another method to gain the right to studentship in open education faculties is the enrollment 
through vertical transfer. Candidates, who want to continue their undergraduate education after graduating 
from Vocational Schools and Open Education Faculty associate degree programs, can enroll through 
vertical transfer. Candidates are required to take the Vertical Transfer Examination conducted by CoASP. 
After the examination, selection and placement procedures are applied as in the Higher Education 
Institutions Examination. 

In the enrollment through lateral transfer, transition to open education associate and undergraduate degree 
programs from formal programs can be performed within the framework of certain legislations and codes 
of practice between associate and undergraduate degree programs in higher education institutions. 

Enrollment within the scope of second university is for the graduates of any higher education 
program and students who are still receiving education. Within the scope of second university, students 
who have graduated from or are studying in an undergraduate program can enroll in associate degree or 
undergraduate programs, and students who have graduated from or are studying in an associate degree 
program can only enroll in associate degree programs. Enrollments to open education undergraduate 
completion programs are carried out by the Council of Higher Education (CoHE, 2018) through central 
placement. 


The Concepts of Active and Passive Learner 
In this section, the concepts of "active learner" and "passive learner" used for open universities are 


discussed. These concepts are not generally used for face-to-face education In Turkey, the students, who 
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have an enrollment in the open education system and perform course enrollment in the relevant education 
term, are called "active learners". Those, who do not perform course enrollment in the relevant term despite 
of having an enrollment and do not have the right to studentship, are called "passive learners". 

Termination of the education by students is only possible with disenrollment according to the higher 

education legislation in Turkey. Disenrollment is only related to the student's own request (petition), 
disciplinary action or period of study. The periods of education are clearly stated in the Law No. 2547. The 
44th clause of the Law No. 2547 is as follows (Legislation Information System, 1981, pp. 5370-1). 
“Students, except for one-year foreign language preparatory class, by starting from the term in which the 
courses related to the enrolled program are provided and regardless of whether they have enrolled for each 
term, have to complete the two-year associate degree programs in maximum four years, four-year 
undergraduate programs in maximum seven years, five-year undergraduate programs in maximum eight 
years and six-year undergraduate programs in maximum nine years. The maximum period of the 
preparatory education is two years.” 
In the related law, the section related to the exmatriculation of students is as follows (Legislation 
Information System, 1981, pp. 5370-1). “Students will not be exmatriculated due to non-payment of the 
contribution or tuition fee within the maximum periods. However, exmatriculation of students can be 
performed due to the decision of the authorized boards of the university and the approval of the Council of 
Higher Education because of non-payment of tuition fees and non-renewal of enrollment for four 
consecutive years.” 

In the last sentence of the clause c and paragraph 3 of the article 44, it is stated that “Open education 
students are not limited to these periods provided that they benefit from their studentship rights”. Regarding 
the legal basis, the disenrollment or exmatriculation process of the students is not performed by the 
institution. 

By force of legal obligations, universities do not perform disenrollment process. If the student does 
not voluntarily disenroll, his studentship continues. This approach is very attractive for learners. They can 
continue their education as an "active learner" at any time they reenroll for any term. It can be stated that it 
is a model applied in the context of the openness philosophy. However, this process increases the number 
of the student group called "passive learner". 

Traditional universities continue to transform some of their face-to-face courses into a form of 
distance education. There are also higher education programs where all courses are performed distantly. 
However, an important problem of distance education is the high dropout rates. At this point, open 
universities can be addressed differently from the traditional universities. In Turkey, Anadolu University is 
among the mega universities in terms of the number of open education students. Examining the number of 
dropout students of open education faculties within the scope of dropout will shed light on the subject of 
“active learners” and “passive learners” of open universities. 


Purpose and Research Questions 


The purpose of this research is to reveal the status of open education active and passive learners within 
the scope of dropout in open and distance learning. In this research, the answers of the following questions 
were searched for. 

1. What is the rate of the number of open education students to the total number of higher 
education students? 

2. What is the change in the total number of open education students by years? 

3. What is the change in the number of active and passive open education learners by years? 

4. What is the change in the number of new enrollment students in open education by years? 


Method 


This study is a descriptive research designed according to screening model. Since the status of open 
education active and passive learners were tried to be analyzed in this research, descriptive screening model 
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was used. Screening models are research approaches that aim to describe a situation existing in the past or 
still as the way it exists. In this research, the numbers of open education active and passive learners and 
their changes by years were examined within the scope of dropout. 

In order to observe the changes in the numbers of students in the higher education system, the data 

in the Higher Education Information Management System of the Council of Higher Education (CoHE) were 
used. The data obtained are limited to the data reached between 2014-2015 academic year and 2018-2019 
academic year. 
In the research, the data included in the CoHE Higher Education Information Management System (Higher 
Education Information Management System, 2019b) and the page of Anadolu University Student Numbers 
(Anadolu University, 2019). The numbers of active and passive open education learners were reached from 
the university web page. The obtained data were analyzed and presented in tables. 

The numbers of Open Education students between the 2014-2015 and 2018-2019 academic years 
were converted to table format. Graphs were used to emphasize the change in the number of students by 
years. In the analysis of the data, numbers and percentages were used and visuality was enabled. The change 
in the number of active learners, change in the number of passive learners, change in the total number of 
students and change in the number of new enrollment students were summarized in tables and graphs. The 
changes in the number of open education students were analyzed by years and findings were created. 


Findings 
In the numerical change in the open education faculties of Turkey, the number of new enrollment 
students is an important factor. In each academic year, new students are accepted based on different 


enrollment types. The total number of the new enrollment students between 2015-2018 are given in Table 
1 by years (Okur, 2019). 


Table 1. The Number of new students enrolled between 2015-2018 to Open Education Faculties 


2015 520.907 
2016 585.484 
2017 486.939 
2018 512.817 


According to the data obtained, it is seen that approximately 520 thousand students enrolled to open 
education faculties between 2015-2018 every year. The concept of New Enrollment is used for all students 
who enroll for the first time within the scope of new enrollment, vertical transfer, lateral transfer, second 
university and degree completion. 

In Turkey, the total student number of face-to-face and open education faculties enrolled in the 
higher education system at associate and undergraduate degree is 7.250.129 by July (Higher Education 
Information Management System, 2019b). The total numbers of open education active-passive learners of 
universities and their rates by the numbers of students in higher education are shown in Table 2. 


Table 2. Percentage of Open Education Students by the Number of Students in Higher Education 


Total Number of Students in Higher Education 7.250.129 
Total Number of Students in Open Education Faculties 3.889.506 
Rate by the Number of Students in Higher Education 53.64% 


The numbers in the table include the total number of active and passive learners of universities. 
The students, who have an enrollment in the open education system and make course enrollment in the 
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relevant education term, are called "active learners". Those, who do not enroll in any course in the relevant 
term even though they have an enrollment and do not have the right to studentship, are called "passive 
learners". By force of the relevant law, the enrollments of students are not deleted due to their education 
periods. In Turkey, there are 3.889.506 open education students enrolled in the higher education system 
(Higher Education Information Management System, 2019b). The total rate of the open education faculty 
students by September 2019 constitutes 53.64% of the higher education system in Turkey. 

Only the numbers of students are included in the Higher Education Information Management 
System. Active or passive information of open education students is not included. The numbers of active 
and passive learners can only be learned through the corporate websites of the universities. It was 
determined that only the numerical data of Anadolu University Open Education System students were 
shared among the universities having open education faculties. The number of passive learners enrolled in 
the Anadolu University open education system is 2.464.412 by March 2019 (Anadolu University, 2019). 
In Figure 1, the distribution of Anadolu University Open Education System students by active-passive years 
is visually summarized. 


Number of Active-Passive Learners 
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Figure 1. Distribution of the numbers of active-passive learners in open education system by years 


In the light of the data obtained, it is seen that the open education passive learner rate in Turkey has 
a tendency to continuously increase by years. This is seen as an important and significant problem that 
needs to be addressed. It is clear that the reasons of students for being passive should be examined in 
institutional terms. 

Enrollment through Second University provides the candidate students with the opportunity to 
enroll in the open education faculty without examination. The graduates of any higher education program 
and students who are still studying can enroll in open education within the scope of Second University 
without taking any examination. It is a very important and attractive enrollment type within the scope of 
lifelong learning. Table 3 shows the number of new enrollment students according to the second university 
enrollment type by years (Open Education Faculty, 2019). 

When Table | and Table 3 analyzed together, it is seen that the majority of the students who newly 
enrolled to higher education system in Turkey are included within the scope of second university. Second 
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university is a type of enrollment that provides lifelong learning for individuals graduating from any higher 
education program. When Table 3 is examined, it is seen that student interest has increased in this context. 


Table 3. The numbers of students enrolled within the scope of Second University between 2015-2018 


2015 135.429 

2016 136.121 

2017 180.996 

2018 224.859 
Conclusion 


In Turkey, students are admitted to the open education system at higher education level under 
different enrollment types. Particularly in the context of lifelong learning, it is seen that enrollment within 
the scope of second university is prominent. According to Table 1, every year approximately 520 thousand 
new students enroll in open education faculties. However, as seen in Figure 1, there is an increase in the 
numbers of passive learners in the face of these high new enrollment figures. Dropout is an important 
problem of the distance education system. 

Mega universities have large rates in terms of student numbers. In these structures, which offer a 
mass education, passive status of students is a situation that needs to be considered. A passive learner can 
become an active learner by enrolling again in the relevant term. This procedure is very simple. It is a 
structure that is very appropriate to the philosophy of openness. Therefore, passive learners in open 
universities need to be handled differently from dropout in distance education practices. Regaining dropout 
students is relatively difficult compared to passive learners. However, open universities can develop ways 
to regain by making strategic moves on passive learners. It can be stated that with different studies, the 
reasons for being a passive learner should be determined and solutions should be developed in this regard. 
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Abstract 


This study demonstrates that longitudinal quantitative analyses outside the established publication record (peer- 
reviewed journals) in shared communities of practice exhibits empirical regularity with persistent patterns of 
lognormal distribution. Bibliometric data was collected from the AECT Conference Proceedings in order to visually 
map communication through co-citation and validate Educational Technology as a scientific specialty. The nature 
(nodal development, seminal works, leading institutions, and researchers at the research front) of educational 
technology was determined. 


Introduction 


Where does scientific specialty begin? How and when do scientific specialties emerge in a discipline? At 
what point in development does a specialty become a field or a discipline in and of itself? Identifying the nature of 
scientific specialties is a fundamental challenge for information science. Morris and Van der Veer-Martens (2008) 
break scientific specialty into three components for the purposes of modeling: researchers, base knowledge, and 
formal literature thereby representing the social, cognitive, and communitive processes in a specialty. When 
scientific specialty is studied in information science, researchers are studied through citation practices, the 
knowledge base is studied through content analyses, and communication is studied through co-citation and 
bibliographic coupling. 

Indeed, co-citation studies of peer-reviewed journal articles recorded in scientific databases, such as Web 
of Science, are the most common methodology used to identify science and make informed purchasing, tenure, and 
science policy decisions; but are they the most effective way to identify scientific specialties? “In a global citation 
analysis, the researcher examines how often a particular publication or a selection of discipline-specific publications 
are cited without regard to the citing authors’ institutional affiliations or geographic regions” (White 2019). Not 
every bit of science is written about and published in traditional scientific journals indexed in traditional databases. 
Conversely, local citation studies focus on the citation habits of users affiliated with a particular institution and 
studies of this nature demonstrate a high percentage of coverage of faculty citations within affiliated libraries 
employing this methodology. In short, we buy or provide access to the science that our faculty write about and cite. 
This phenomena may not provide insight about emerging specialties. Further, with the expanding opportunities to 
publish outside peer-reviewed journals, especially for technology related fields, a general absence of specialty 
emergence may occur within a local library collection or global indexes. A method to study scientific specialties 
that coalesce outside the established publication record should be considered. 


Theoretical Background 


Kuhn’s seminal work, The Structure of Scientific Revolutions, emphasized the importance of community in 
the development of science in that paradigmatic development occurs with shared practices of inquiry. Science, in its 
purest iteration, is a shared social practice and depends on a shared professional context or “selection environment” 
(Leydesdorff, Wagner, & Bornmann, 2018). “Research specialties consist of relatively small self-organizing groups 
of researchers that tend to study the same research topics, attend the same conferences, publish in the same journals, 
and also read and cite each other’s research papers” (Morris & VanderVeer-Martens, 2008, p.213). 
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According to Rons (2018), “Sufficient bibliometric focus at the specialty level requires an aggregation of 
publications that is more fine-grained than the broad subject categories grouping interrelated journals that form the 
backbone of commonly used global publication and citation indexes” (p. 114). Global publication and citation 
indexes overlook some publications until such time that the publication has sufficient global appeal. Conference 
proceedings, which are not often indexed in global indexes, provide a set of publications that focus on the 
interactions of a self-organizing research community that are often overlooked but provide a sufficient oeuvre for 
the identification of scientific specialties in that conference proceedings provide a view at the convergence of 
community. 

The goal of this study was to conduct bibliometric analyses of a non-traditional oeuvre from a self- 
organizing group of researchers--conference proceedings--in order to inform about its nature as a specialty by 
measuring its growth through lognormal distribution and visually mapping its development through network 
analysis as a knowledge domain. 

Methods 


Scientific specialties form communities of practice most often with a convergence of the community in 
technology sectors through conference attendance. The Association for Educational Communications and 
Technology (AECT) is an international organization of instructional/educational technologists and researchers that 
has published its annual conference proceedings since 1979. These conference proceedings provide 35 years of 
published papers in this specialty that were not indexed in traditional journal databases and therefore provide an 
overlooked knowledge domain worthy of study for the identification of the nature of this scientific specialty. 

Bibliometric indicators were extracted and collected into custom databases for truncation into 
Characteristic Scales and Scores (CSS), co-citation analyses, and multidimensional graphing. Mean citation rates 
and distribution were calculated in 5-year sets. The application of the CSS method took into account the full set of 
citation counts where all authors were considered equal. Lognormal distribution was iteratively generated following 
a specific parametrization where the CSS algorithm was applied to each sample and the values were identified at the 
point of convergence. 

Co-citation was tracked in custom databases in 5-year dispersions in order to conduct visual network 
analysis. Gephi 0.9.2 software was used to create multidimensional graphs to identify developmental and 
incremental change within the knowledge domain with the purpose of understanding the developmental pattern of 
the specialty. 

Results 


The researchers, seminal works, and the formal literature in each data set were identified and exhibit 
indicators for decision making. Table | represents the most cited authors within the thirty-year studied span and 
the knowledge base of Educational Technology. Gagne, Hannafin, Jonassen, Dwyer, and Reigeluth were the most 
cited authors over the 30 year period. 


Table 1. Authors Representing the Knowledge Base 


1979-1984 1985- 1990- 1995- 2000- 2005- 
1989 1994 1999 2004 2009 
Dwyer Tennyson Jonassen Gagne Jonasson Jonassen 
Witkin Gagne Gagne Jonassen Reiguluth = Mayer 
Salomon Clark Hannafin Hannafin Mayer Garrison 
Gagne Hannafin Reigeluth Keller Duffy Anderson 
Snow Salomon _ Ross Reigeluth Hannafin Ertmer 
Paivio Merrill Tennyson Collins Keller Sweller 
Cronbach Anderson Salomon Duffy Gagne Reigeluth 
Goodenough Snow Merrill Brown Dick Archer 
Clark Reiguluth Mayer Dick Bandura Hannafin 
Winn Witkin Keller Dwyer Collins Bandura 
Levin Dwyer Johnson Ross Clark Keller 
Fleming Ross Dwyer Salomon Ertmer Pintrich 
Simonson Kulik Clark Klein Wenger Schunk 
Levie Mayer Johnson Johnson Bonk Wenger 
Ausubel Winn Wittrock Johnson Moore Bonk 
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Travers Kulhavy — Carrier Merrill Gunawarde Duffy 


na 
Lamberski Rakow Briggs Hooper Richey Klein 
Merrill Heinich Morrison Sullivan Sweller Vygotsky 
Karp Levin Rieber Wilson Becker Moreno 
Berry Rieber Collins CTGV Carey Gunaward 
ena 
Mayer Levie Brown Slavin NCES Brown 
Oltman Briggs Dick Briggs Schunk Zimmerm 
an 
Allen Canelos = Kulik Morrison Wilson Bruner 
Briggs Phillips Winn Wager Salomon Bransford 


Seminal Works are represented in Figure 1. The most influential reference work of the period studied was 
Principles of Instructional Design by Gagne and Briggs with over a twenty-five-year span of influence. Two other 
titles influential over a twenty-year span were The Conditions of Learning by Gagne, and Instructional Design 
Theories and Models: an Overivew of Their Current Status by Reigeluth. 


Figure 1. Timeline of Seminal Works and Their Span of Influence 
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Figure 1. Timeline of the most cited Reference Works presented in linear scale over the division of the 
data subsets with each Reference Work title orb indicating span of influence. Created with Online 
Diagram Software & Visual Solution | Lucidchart. Copyright 2018 by www.lucidchart.com. 
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Table 2 represents the most cited academic publications within the thirty-year span of Educational 
Technology represented in this knowledge domain. 


Table 2. Most Cited Academic Publications (Formal Literature) 


All Journal Citations 


Variable N=48,063 
J OF EDUCATIONAL PSYCHOLOGY 2075 
EDUCATIONAL TECHNOLOGY 1471 
ETR&D 1393 
REVIEW OF EDUCATIONAL RESEARCH 930 
ECTJ 657 
AV COMMUNICATIONS REVIEW 602 
J OF EXP PSYCH: HUMAN LEARNING & MEMORY 495 
EDUCATIONAL RESEARCHER 487 
J OF COMPUTER BASED INSTRUCTION 462 
AMERICAN EDUCATIONAL RESEARCH JOURNAL 332 
J OF INSTRUCTIONAL DEVELOPMENT 309 
THE AMERICAN J OF DISTANCE EDUCATION 285 
INSTRUCTIONAL SCIENCE 283 


Lognormal Distribution 

Characteristic Scales and Scores (CSS) offered a straightforward measure for benchmarking the citation 
performance of individual authors in relation to their peers and the overall population based on a common 
framework of algorithmically constructed performance classes. This methodology relied on “a recursive procedure 
of iteratively truncating a sample according to mean values from the low-end up to the high-end” (Glanzel 2011, 
p.42). While citation studies have relied on the Pareto Distribution (80/20 rule) since the 1960’s, a recent study by 
Viiu (2018), found that “irrespective of scientific field and citation window, CSS tend to uncover an extraordinarily 
stable distribution of papers across predefined classes of citedness. Virtually all fields of science are shown by CSS 
to be fundamentally similar in that they share an approximate 70-21-6-3% distribution (Viiu, 2018, p.402) Evans, 
Hopkins, and Kaube (2012) confirm this claim at the level of a specific research institute, at the sub-level of 
departments, but also for data from the arXiv eprint archive. Perianes-Rodriguez and Ruiz-Castillo (2016) also 
verify this distribution at as a universal claim at the institutional level. While three performance classes were 
employed in this study, CSS identified a lognormal distribution. 


Table 3. Citations by Dataset 


Subset 5 or more Citations 2to4 1 Citation 
Citations 
1979-1984 10(5%) 37(20%) 141(75%) 
1985-1989 16(7%) 45(19%) 176(74%) 
1990-1994 16(4%) 99(25%) 278(72%) 
1995-1999 13(3%) 106(23%) 345(74%) 
2000-2004 27(3%) 256(28%) 635(69%) 
2005-2009 22(3%) 174(24%) 518(73%) 
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Network Analysis 

According to Citron and Way (2018), “Co-authorship networks are a measureable representation of the 
communities that assemble in order to work in a particular area of research (p. 181). In network theory, centrality 
focuses on the interaction between individual participants within a network. Between-ness Centrality (BC), the most 
prominent measure of centrality in network analysis, measures the relative number of times that a node (individual 
participant) is part of the shortest distance (the co-citation connection) between nodes (individual participants) in a 
network (Leydesdorff, Wagner, & Bornmann, 2018). Figures 2-8 represent a diagrammatical representation of the 
authors in terms of influence during each data set employ Between-ness Centrality to display co-citation connections 
and, taken as a series, demonstrate growth. A topological transition was apparent between datasets demonstrating 
the communication interactions of a mature scientific specialty or field. 


Figure 2. Data Subset 1979-1984 Cocitation Network Analysis of the Most Influential Authors 


STALEY can@eu. 


AUSBURNLJ 


Ha@ev 


MERRILL 


Figure 2. A bibliometric multi-dimensional map of the author cocitations from the Data Subset 
1979-1984 from the Conference Proceedings of the AECT. Visualized with Gephi 0.9.2 software 
employing the Yifan Hu algorithm. 
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Figure 3. Data Subset 1985-1989 Cocitation Network Analysis of the Most Influential Authors 


Figure 3. A bibliometric multi-dimensional map of the author cocitations from the Data Subset 1985- 
1989 from the Conference Proceedings of the AECT. Visualized with Gephi 0.9.2 software employing 
the Yifan Hu algorithm. 
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Figure 4. Data Subset 1990-1994 Cocitation Network Analysis of the Most Influential Authors 


Figure 4. A bibliometric multi-dimensional map of the author cocitations from the Data Subset 1990-1994 
from the Conference Proceedings of the AECT. Visualized with Gephi 0.9.2 software employing the Yifan 
Hu algorithm. 
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Figure 5. Data Subset 1995-1999 Cocitation Network Analysis of the Most Influential Authors 


Figure 5. A bibliometric multi-dimensional map of the author cocitations from the Data Subset 1995-1999 
from the Conference Proceedings of the AECT. Visualized with Gephi 0.9.2 software employing the Yifan Hu 
algorithm. 
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Figure 6. Central Embryonic Cluster of Data Subset 2000-2004 Cocitation Network Analysis of the 
Most Influential Authors 


Figure 6. A bibliometric multi-dimensional map of the author cocitations from the Data Subset 
2000-2004 from the Conference Proceedings of the AECT. Visualized with Gephi 0.9.2 software 
employing the Yifan Hu algorithm. 
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Figure 7. Data Subset 2005-2009 Cocitation Network Analysis of the Most Influential Authors 


Figure 7. A bibliometric multi-dimensional map of the author cocitations from the Data Subset 2005-2009 
from the Conference Proceedings of the AECT. Visualized with Gephi 0.9.2 software employing the Yifan Hu 
algorithm. 
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Conclusion 


The nature of scientific specialties can be identified in knowledge domains outside of the traditional 
publication record of global indexes. “Disciplinary communities have been described as tribes each with its own 
norms, categorizations, bodies of knowledge, sets of conventions, and modes of inquiry, which compromise a 
recognizable culture” (Hyland & Salager-Meyer, p. 311). This study provided on one level a simple working model 
of specialty identification at the convergence of community that included the network of researchers, the base 
knowledge, and the specialties’ formal literature. 

On another level, specialty identification was confirmed by the lognormal distribution of Characteristic 
Scales and Scores. Virtually all fields of science are shown by CSS to be fundamentally similar and the presence of 
CSS 70-21-9% distribution of citation counts within the conference proceedings as the knowledge domain 
representing Educational Technology confirms the presence of a scientific specialty or field. 
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Abstract 


There are many ways instructors and instructional designers can design and develop multimedia for online 
courses. This research looked to define evidence-based best practices for the design of multimedia content that also 
enhanced social presence in distance learning and online environments. The study asked 596 participants to 
consider the perspective of online students and select one of five videos that they would prefer viewing during their 
online course. These participants included a diverse sample of students, faculty, administrators, researchers, and 
instructional design and support staff. The findings indicate that 91.7% of participants preferred the two variations 
that included video of the instructor and the instructor’s presentation slides at the same time. Among the reasons 
why, participants responded, “this is more like an actual classroom experience” and “T like being able to see the 
instructor.” These results also indicate the importance for instructors and instructional designers to consider the pre- 
production and post-production implications and resources required to develop content in these formats. 


Introduction and Background 


Instructors and instructional designers have many tools at their disposal to develop multimedia content for 
their online classes. This research study sought to better understand how specific combinations of video and content 
could help foster social presence. Social presence is the ability to use technology to create and foster interpersonal 
relationships in learning environments between instructors and students (Short, Williams, & Christie, 1976). Social 
presence in instructional message design can be used to enhance motivation, which can lead to learning 
effectiveness and student retention (Fleming & Levie, 1993; Ramlatchan, 2019). Multimedia is the inclusion of 
several means to communicate information integrated thoughtfully in the same presentation to create or enhance the 
learning experience (Clark & Mayer, 2016). This study combined applications of video, multimedia learning and 
design, and their potential impact on social presence to help inform best practices in instructional message design. 
The design of the modules used in this study presented multimedia to viewers using several combinations of 
recorded video and presentation slides with text and graphics. The goal was to determine which design 
combinations resonated best with students, especially in terms of enhanced social presence. This study’s specific 
research question sought to explore which design was preferred by students, faculty, and instructional design and 
support staff and why? 


Research Design 
This study asked viewers to compare five variations of a multimedia presentation and to select which design 


they preferred. Each of the five designs are currently used in the host university’s distance learning courses, online 
programs, and are representative of existing course content. An experienced presenter was recorded in one of 
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Participants viewing the “instructor-only” option were 
only able to see the instructor, while volume was turned 
down during the data collection, participants were asked 

to assume that the narration was the same for all five 
options 


Participants viewing the “slides-only” option were only 
able to see the instructor’s PowerPoint slides, they would 
not see the instructor 


Participants viewing the “video-switching” 
option saw the video alternate between a view 
of the instructor and a view of the PowerPoint 

slides (the slides were viewable long enough 
for participants to read each slide) 


Participants viewing the “dual-windows” option could see 
the instructor in a window in the upper left and see the 
PowerPoint slides in a larger window on the right 


Participants viewing the “layered-video” option could see 
the instructor as a layer of video in the foreground and 
they could see the PowerPoint slides as a second layer of 
video behind the instructor 


What is Social Networking? 


What is social media? 
What is social networking 
technology? 

Connect with friends, 
friends of friends 
Examples: MySpace and 
Facebook 

First technology milestone: 
written alphabet 
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Figure 1. Each participant was able to compare each multimedia design and select their preference (modified from 


Ramlatchan & Watson, 2019) 
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the university’s audio and video production studios. The instructor presented a 20-minute module on social media 
and social networking intended for an online “communications” or an “introduction to technology” course. Thus, the 
instructor, instructor video, and the content presented in the PowerPoint slides were kept consistent throughout the 
study. 

Data was collected at Old Dominion University’s 2018 Faculty Summer Conference, from online and 
virtual classroom students also at Old Dominion University, from the University Reception at the 2018 AECT 
(Association for Educational Communications and Technology) conference, and during the 2018 SACSA (Southern 
Association for College Student Affairs) conference. The result was a diverse sample of 596 participants that 
included students, faculty, administrative staff, and instructional design and support staff across a wide range of 
ages, academic experience, and subject matter areas. The five multimedia designs presented to participants included 
an instructor-only, a slides-only, a video-switching, a dual-windows, and a layered-video variation (see Figure 1). 
These five multimedia designs are currently in use in some form at the host university in online classes and 
programs and represent examples of what online students enrolled in those programs would see. The instructor-only 
version only showed video of the instructor’s camera. The slides-only version only showed the instructor’s 
PowerPoint slides. The video-switching version showed the instructor’s PowerPoint slides long enough for a 
student to read a slide, then the video switched back to show the instructor until the next slide. The dual-windows 
version showed the instructor video as a smaller window on the screen and showed the instructor’s PowerPoint 
slides as a larger window. The layered-video version used a black background with the slides as a second layer of 
video over the background and the instructor as a third layer of video over both the background and slide layer. 

All five designs were played as 20-minute videos on a continuous loop on five identical laptops (see Figure 
2). The five laptops were numbered and setup side-by-side on a 6-foot table with a ballot box paired with each 
laptop, this arrangement allowed participants to simultaneously view and compare all five designs. When potential 
research participants approached the table they were given a pen, a clipboard, a 3-inch x 5-inch card to write why 
they made their selection, and asked “if you were an online student which of these videos would you prefer?” The 


Figure 2. Each of the five options were played on a loop on five identical laptops, participants wrote their 
reasoning for selecting their preference on a 3x5-inch card and dropped the card in the appropriate ballot box 
(the “video-switching” option was played on the center laptop in this example, and was showing the slides when 
this picture was taken, it would periodically alternate between showing the slides and video of the instructor 
during data collection). 
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contributing students, faculty, administrators, and staff indicated the design they preferred, wrote a short reason why 
they made that selection, and dropped their card into the appropriate ballot box. Participants received either a t-shirt 
or a plush mascot keychain for their feedback. The data collection process took approximately 30 to 60 seconds to 
complete. 


Results and Discussion 


The results of this experiment were surprisingly consistent, especially given the four diverse samples used 
in data collection. The 2018 Faculty Summer Conference group consisted of students but was mostly a group of 
instructional design staff, administrators, and faculty at the host university (n = 158). Of this group, 79.1% preferred 
the layered-video, 14.6% preferred the dual-windows, 5% preferred the video switching, 1.3% preferred the slides- 
only, and 0% preferred the instructor-only. 

The virtual classroom and online student group consisted of approximately 85% students and 15% faculty 
and staff enrolled in, or teaching, or supporting classes at the host university (n = 176). Of this group, 60.2% 
preferred the layered-video, 31.3% preferred the dual-windows, 5.1% preferred the video switching, 2.3% preferred 
the slides-only, and 1.1% preferred the instructor-only. 

The 2018 AECT conference group was a very diverse range of students, instructional designers, 
administrators, teaching faculty, and researchers from a wide range of private, public, large, small, regional, 
national, international, online, and traditional on-campus colleges and universities (n = 138). Of this group, 70.3% 
preferred the layered-video, 18.1% preferred the dual-windows, 6.5% preferred the video switching, 2.9% preferred 
the slides-only, and 2.2% preferred the instructor-only. 

The 2018 SACSA conference group also represented a diverse number of students, faculty, and staff in a 
wide range of student support disciplines from a number of colleges and universities in the southeast region of the 
United States (n = 124). Of this group, 75% preferred the layered-video, 18.5% preferred the dual-windows, 5.6% 
preferred the video switching, 0.8% preferred the slides-only, and 0% preferred the instructor-only. 

In total, there were 596 participants, 70.6% preferred the layered-video, 21.1% preferred the dual-windows, 
5.5% preferred the video switching, 1.8% preferred the slides-only, and 0.8% preferred the instructor-only (see 
Table | and Figure 3). Based on observations during data collection it is estimated that 50% of the total participants 
were active students, while the other half were faculty, staff, and administrators. However, the perspective of 
faculty, staff, and administrators are important to consider as not only were these participants likely former college 
students at one time, they also fall into the adult learning demographic for prospective future graduate students. 
Additionally, the researcher has found no significance difference when comparing pilot data collected from 
confirmed students and pilot data collected from faculty, staff, and administrators in terms of similar social presence 
and multimedia studies (Ramlatchan & Whitehurst, 2019). 


Table 1. The number and percentage of participants who selected each option 


Multimedia Design # of participants who % of participants who 
Option selected that option selected that option 


SlidesOnly | unt 


Video Switching | 33, | 5% 


Dual Windows | 126 21.1% 


Layered Video | 421 70.6% 


596 
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What is Social Networking? 
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Figure 3. The layered-video option was the most preferred multimedia design, followed by the dual-windows 
variations (for a combined 91.7%), this result would appear to indicate the overall preference for the ability to see 
both the instructor and the instructor’s slides during online courses that include video. 


The preference for seeing both the instructor and the instructor’s slides is consistent with other social 
presence research that found similar results (Ramlatchan & Watson, 2019). Qualitative feedback for these two 
options often included a variation of “this is more like an actual classroom experience” and “T like being able to see 
the instructor.” Both the quantitative data and the qualitative feedback on their 3x5 cards indicated the importance 
of social presence or the ability to see an authentic instructor. 

The most preferred options (the overall preference for layered-video and the dual-windows options for a 
combined 91.7%) were also the most resource intensive to produce. Interestingly, the options that likely require the 
least amount of time and effort to produce, were also the options least preferred by participants. The slides-only 
version is a very common online presentation technique as there are many programs that can add instructor audio 
voiceover to screenshots, screen grabs, or other means of slide capture. The ‘talking head’ of the instructor-only 
version is also very ubiquitous, easy to create, and was also among the least preferred by participants in this study. 
The dual-windows option is created by a telepresence recording platform from Cisco Systems that while automating 
the recording of web conferencing classes, also requires a significant infrastructure investment. The layered-video 
variation was by far the most preferred, but also requires the most overall time and effort to produce. A post- 
production time and effort investment must be made as a video editor or producer has to create this presentation by 
making a new video that is a composite of the virtual background, the slides, and the audio and video recording of 
the instructor. However, the result of this production effort is a presentation format that retains as much as possible 
the social presence aspects of the face-to-face classroom environment for online students. 
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Conclusion 


The purpose of this study was to better understand the impact of multimedia message design on social 
presence based on the preferences of a large diverse sample of students, faculty, and staff among several colleges 
and universities. Participants viewing the five options were simply asked “if you were an online student which of 
these videos would you prefer?” The results indicate that participants consistently selected the multimedia designs 
that allowed them to see both the video of the instructor and the PowerPoint slides for the full duration of the 
presentation. This consistency was present when traditional students (approximately 50% of the total 596 
participants) provided feedback, as well as when instructional design staff, instructors, and administrators (who also 
fall within the host university’s adult learner demographic) provided feedback. The layered-video version was by 
far the most popular for several reasons which include the design’s approximation of the face-to-face classroom 
experience and the maximization of social presence. These results can also likely be generalized and applied to a 
diverse set of learning environments, across many fields of study, at many colleges and universities. Online students 
prefer multimedia designs that enhance social presence, such as multimedia designs that allow them to see both their 
instructor and their instructors’ presentation at the same time. This instructional message design best practice can be 
used to enhance motivation, retention, and overall learning effectiveness in online courses and programs. 
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Abstract 


What are the benefits of interactive, two-way video and web conferencing in online classes in terms of 
social presence? This study compared four presentations, two were synchronous and were delivered in real-time and 
two were pre-recorded and delivered asynchronously. Student volunteers (n = 162) were organized into four 
treatment groups. The first treatment group participated from an immersive telepresence, video conferencing 
system, the second group participated via online web conferencing, the third group viewed a recording of the 
telepresence group’s meeting, and the fourth group viewed a recording of the web conferencing group’s meeting. 
The use of the telepresence system improved participants’ sense of instructor immediacy and social presence, 
followed by the perception of the web conferencing group, while the asynchronous viewers reported a lower sense 
of connectiveness. The data analysis also indicated no significant difference among the treatment groups in terms of 
extraneous cognitive load. The results indicate the social presence benefits of real-time engagement, active learning, 
and suggests that instructors and instructional designers should consider the integration of synchronous activities in 
their online classes. 


Introduction 


We know that maintaining retention and ensuring learning effectiveness in online environments is a 
challenge for distance learning and distributed education programs. One means to support both motivation and 
learning effectiveness is to create learning environments that foster and enhance social presence. Social presence in 
the context of education is the student’s perception that they are communicating and engaging with an authentic 
instructor (Short, Williams, & Christie, 1976). Social presence in this study was explored in terms of instructor 
immediacy, or the verbal and nonverbal communication effectiveness of the instructor from the perspective of 
online students (McCroskey & Teven, 1999; Violanti, Kelley, Garland, & Christen, 2018). The social presence 
implications of instructor immediacy has been explored in a number of other multimedia studies, however, these 
studies focused on pre-recorded, scripted instructor presentations (Jayasinghe, Morrison, & Ross, 1997; Ramlatchan 
& Watson, 2019). The present study has a similar research design, though instead now includes two treatments with 
a live, real-time instructor that encourages student interactivity and asking questions. 

Instructional message design is the purposeful, thoughtful, and evidence-based development of 
instructional material, content, or other vehicle of instruction and can influence motivation, learning effectiveness, 
and retention (Fleming & Levie, 1993; Ramlatchan, 2019). The ideas of social presence and message design can be 
combined with the heuristics of multimedia learning to further improve overall learning effectiveness. Multimedia 
learning theory suggests using words and pictures, reducing extraneous information, and connecting new concepts to 
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previously learned concepts in instructional designs (Clark & Mayer, 2016; Mayer, 2009). Multimedia learning 
theory builds upon cognitive load theory, which suggest that extraneous cognitive load should be removed or 
reduced as much as possible from message designs to improve learning efficiency and effectiveness (Paas & 
Sweller, 2014). 

In the context of this study, the multiple media used in the instructional message design includes 
synchronous or asynchronous online visuals of both the instructor and the instructor’s presentation as well as the 
narrative voice of the instructor. While there is an established body of knowledge on the use of multimedia learning 
theory, there has not been significant research into the area of live, interactive, synchronous video. This study 
sought to explore the connection between multimedia design, social presence, and interactive video. Specifically, 
when the instructor and the subject matter are kept constant, how does the inclusion of live engagement in the 
multimedia design impact social presence in terms of student perceptions of instructor immediacy? This research 
project also gauged the impact of each presentation on student perceptions of extraneous cognitive load. 


Research Questions 


The problem this research sought to address was the lack of quantitative experimental results when 
comparing a student’s ability to interact with an instructor and the impact on social presence in terms of instructor 
immediacy and the impact in terms of any added extraneous cognitive load: 


Research Question 1: What impact will variations of synchronous and asynchronous online multimedia 
presentations have on student perceptions of instructor immediacy? 


Research Question 2: What impact will variations of synchronous and asynchronous online multimedia 
presentations have on student perceptions of extraneous cognitive load? 


Research Design 


This study employed four treatment groups to explore interactive multimedia and instructor immediacy (see 
Figure 1). The same instructor presented the same subject matter with the same PowerPoint presentation in each 
group. The slides were designed to present the content in a clear, concise manner, with a concerted focus to avoid 
extraneous distractions. The instructor presented a 20-minute module on “Transcendentalism, Romanticism, and 
Regionalism” from her online American Literature course. There were four specific times in the mini-lecture where 
the presenter specifically asks a question to engage and solicit thoughts and feedback from participants. The first 
treatment, the telepresence group, employed two telepresence equipped classrooms. Telepresence in the context of 
this study is the practical application of immersive video conferencing, such as using a Cisco [X-5200 or similar 
immersive system which includes high resolution video, multiple large, high-definition displays, and high-fidelity 
audio. The instructor presented from one classroom, volunteers participated from another classroom. The students 
and the instructor could see and hear each other in real-time in each classroom and the students could also see the 
PowerPoint slides. Students in the second treatment, the web conferencing group, participated online via web 
conferencing, with the instructor still in the same telepresence classroom as the first treatment group. Each student 
could see the instructor and see the PowerPoint slides. The third treatment group viewed a recording of the 
telepresence group’s meeting. The fourth treatment group viewed a recording of the web conferencing group’s 
meeting. These two groups could also see the instructor and the slides. 

Volunteers for this study were students enrolled at a mid-size, metropolitan, public university. A link to 
participate in the study was included in a university events and announcements email sent daily to all students; the 
only requirement to participate was the ability to attend the presentation in the telepresence classroom or have a 
reliable Internet connection. The study continued until at least 30 students had participated in each treatment group. 
The social presence component of instructor immediacy was measured using the McCroskey’s Source Credibility 
Measure (McCroskey & Teven, 1999). Cognitive load was measured using the NASA Task Load Index (Hart, 
2008; Hart & Staveland, 1988). These two survey instruments, and an additional item asking them to enter in any 
additional comments, were accessible to participants as a survey after they viewed the presentation. Each student 
only participated in one treatment group and each were given a free t-shirt for their time and feedback. 
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Figure 1. Volunteers in this study participated in one of four different treatment groups, two were delivered live and 
allowed for real-time interaction with the instructor and two were pre-recorded (modified from Ramlatchan & 
Whitehurst, 2019). 
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Results 


A one-way univariate Analysis of Variance (ANOVA) was conducted to determine the effect of the four 
presentation methods on the mean score of participants on the Nonverbal Immediacy Behaviors Index section of the 
survey. The ANOVA indicated a significant difference between treatment groups, F(3,156) = 5.8, p <.01 (see Table 
1). A follow-up Tukey HSD test was conducted to determine the distinction among the groups, which showed a 
statistically significant difference between the telepresence group and the three other treatments (see Figure 2). 


Table 1. Nonverbal Immediacy Behaviors Index Measure of each Treatment Group 


95% Confidence 
Treatment n Mean (SD) Interval 
Synchronous telepresence video conference 33 5.03 (.53)* [4.85, 5.21] 
Synchronous web conference 38 4.54 (.72) (4.31, 4.77] 
Asynchronously viewed telepresence video conference 51 4.51 (.67) [4.33, 4.69] 
Asynchronously viewed web conference 37 4.42 (.73) [4.18, 4.66] 


Note. Nonverbal Immediacy measured on a | to 6 scale, with 6 being the most immediate. 
* A significant difference was found between this treatment and the three other treatments, p < .05 


Instructor Immediacy 


¢ Participants rated instructor immediacy higher in 
the Telepresence treatment 


45 


Asynchronously viewed Asynchronously viewed web 
telepresence video conference conference 


ynchronous web conference 


Synchronous telepresence vide 
conference 


Figure 2. A one-way ANOVA and Tukey HSD post-hoc analysis indicated a significant difference in participant 
perception of immediacy, or the communication effectiveness of the online instructor. Participants in the live 
telepresence classroom treatment felt a high degree of social presence. 


A one-way univariate Analysis of Variance (ANOVA) was also conducted to determine the effect of the 
four presentation methods on the mean score of participants on the NASA Task Load Index section of the survey 
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(see Table 2). While a pattern may appear to forming, the ANOVA indicated no significant difference between 
treatment groups in terms of participant perception of extraneous load, F(3,158) = .32, p = .81 (see Figure 3). 


Table 2 


NASA Task Load Index (TLX) Measure of each Treatment Group 


95% Confidence 
Treatment n Mean (SD) Interval 
Synchronous telepresence video conference 33 2.35 (1.62) [1.83 to 2.87] 
Synchronous web conference 39 2.45 (1.56) [1.96 to 2.94] 
Asynchronously viewed web conference 39 2.45 (1.97) [1.83 to 3.07] 
Asynchronously viewed telepresence video conference 51 2.72 (1.69) [2.26 to 3.18] 


Note: Extraneous load measured via the NASA TLX was measured on a 0 to 10 scale, with 10 being the most 
demanding, stressful, or cognitively distracting. 
No significant differences were found between these treatments, p < .05 


Perceived Extraneous Load 
¢ Participants rated extraneous load statistically equivalent =o | 


N 


Synchronous telepresence video Synchronous web conference Asynchronously viewed web Asynchronously viewed 
conference conference telepresence video conference 


Figure 3. A one-way ANOVA indicated a no significant difference in participant perception of extraneous load 
when measured with the NASA TLX (unweighted), although there does appear to be a pattern indicating the ease of 
use of the classroom environment. 


The online survey used in this study included a write-in box for other participant comments. This 
qualitative data collection yielded several interesting responses. One participant during the synchronous web 
conferencing treatment reported: 

I think the instructor in this case covered things very well and seemed, in hindsight, to give a lot of time to 
students for questions and comments. I can see classes like this feeling a lot more communal to distance learning 
students, giving students the opportunity to discuss and "be together" all in one place at the same time. 
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This statement appears to highlight the importance of keeping students engaged. Another participant 
during the synchronous telepresence treatment echoed this response: 

The interaction with the instructor was super helpful due to the fact that we were actually able to engage in 
conversation and when any of us had questions, she’d be more than willing to answer and explain information 
further. 

This feedback would tend to indicate some of the advantages of a synchronous online tool in terms of 
allowing the instructor to guide a discussion beyond a scripted presentation. However, even participants in the 
asynchronous treatment groups appeared to appreciate the instructor’s desire to engage with students. For instance, 
one participant during the recorded asynchronous telepresence treatment commented: 

The instructor was very well paced through the presentation; questions were asked throughout the 
presentation that were thought provoking and let the discussion evolve with the participants. 


Conclusions 


The significant contribution to the existing body of knowledge on the subject of multimedia, social 
presence, and online learning was the use of interactive communication in two of the four treatment groups. This 
allowed for a comparison between live interactivity and pre-recorded, asynchronous multimedia presentations, using 
the same content and instructor. The results of this study indicate that participants rated the instructor highest in 
terms of immediacy during the live interactive telepresence treatment. A pattern in the data indicated that both live 
treatment groups outscored the two recorded video viewing groups in terms of perceived instructor immediacy. This 
finding suggests and helps confirm that students feel a high degree of social presence when they are able to engage 
in real time with their instructor. This study also indicated that while the peripheral distractions of the classroom 
environment was rated lower, neither of the presentation formats appeared to statistically significantly differ from 
each other in terms of cognitive load, especially extraneous cognitive load. These findings could indicate that the 
design of the classroom, the design of the web conferencing application, and the design of the video playback 
interface did not introduce a significant level of extraneous cognitive load. The instructional message design of 
these learning environments did not appear to negatively impact learning effectiveness from this perspective. 

Feedback from participants who viewed the recordings and commented that they appreciated the instructor 
asking and answering questions and engaging with the live students were interesting given that they themselves 
were unable to ask questions and participate in that discussion. In effect, it appears that both synchronous and 
asynchronous students benefited from the instructor actively engaging with the live audience. While this finding 
makes intuitive sense in terms of social learning and social presence, a future study could remove the question and 
answer periods from the recording and try a new series of treatments to confirm this conclusion. Another avenue for 
future research could be the use of eye-tracking techniques to more directly measure extraneous cognitive load as 
compared to the indirect self-reporting used in this study. Yet another series of experiments could vary the 
instructor, subject matter, and include the use of whiteboards or writing on digital tablets to further explore how 
multimedia presentation design can impact social presence in our online learning environments. 

To increase social presence in their online classes, and thus increase learning effectiveness and motivation, 
instructors and instructional designers should consider the use of synchronous tools in their online instructional 
designs. The ability for students to see and hear their instructor, live and in real time, and the ability to ask 
questions, have a discussion, and actively engage in learning can create and enhance positive environments for 
social presence in online classes and programs. 
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Abstract 


This poster reports an exploratory study about the process or the quality of the search online information, 
including how the information was found and “read” (or “seen”). Inspired by the eye-tracking technology and the 
idea of “eye-mind hypothesis,” the study explores students’ online information searching/browsing behaviors and 
what eye movement data can provide to better understand Web searching/browsing behaviors, and therefore, to 
inform practitioners (i.e., instructional designers, librarians, and information specialists etc.) for better design. 
Keywords: information seeking behavior, scan path; eye tracking 


Introduction 


In the information age, students constantly search for information on the internet. Sometimes a search for 
information to specific inquiries is successful; sometimes it is not. Even with success, scant information is available 
about the process or the quality of the search, including how the information was “read” (or “seen”’), not to mention 
whether the reader engaged with the information that was seen or read. Understanding how users perform online 
information searching is a pervasive interest of librarians, information specialists, and instructional designers. For 
example, researchers have been focused on how visual design or presentation/display format was associated with 
learning performance, reading comprehension, and decision making. (Agostinelli et al., 2012; Bettman & Kakkar, 
1997; Biehal & Chakravarti, 1982; Tomita, 2017; Zikmund-Fisher et al., 2016). While conventional user research 
methods are effective in obtaining users feedback on usability and their experiences, they have several key 
limitations. For example, traditionally one can verify whether the users actually “look at” the information on the 
present screen (or Web pages) by asking the participants follow-up interview questions and retrospective analyses of 
their online behaviors or to engage in self-report/think-aloud activities as they search. However, the latter approach 
may interrupt participants as they engage in resource intensive search behaviors, while the former methods rely on 
limited human memory. With eye tracking data (i.e., scan paths, fixation counts, and fixation duration), the 
researcher can explore a user’s searching and ‘reading’ behaviors by accessing the visual path track. Therefore, this 
study investigates online information browsing behavior with deeper exploration by incorporating eye-tracking data 
into conventional user research methods. 


Background 


The research detailed here was driven by a previous usability study, which the instructional designer 
accidently found that some users responded with the ‘wrong’ answer to a common library inquiry when they all 
arrived at the ‘target’ pages where information was located. A pilot study (Sun, Sheu, & Tsai, 2018), as part of a 
large project, was meant to be a test round for the protocol and the tasks were serve as warm-up tasks. Therefore, the 
tasks were designed to be simple and easy to complete. The tasks were to find answers from a selected website for 
two simple questions derived from the commonly asked questions at the library settings (e.g., library hours, 
directions, and eligibility of ‘membership’ etc.). The two questions were: (1) what’s the library hours for Tuesday? 
and (2) Can a 15-year old high school student apply for a library card? 

In the pilot study with eight participants, all participants completed both tasks/questions within 90 seconds, 
meaning that all participants were efficiently locate the information on the website and turned in the task sheet with 
their answers. The total number of Web pages visited was less than six for both tasks, excluding the home 
page/starting page. It was another indicator for efficient performance. All participants fulfilled the first inquiry 
correctly, but surprisingly three out of the eight respondents got the second question wrong even though they all 
successfully ‘locate’ the information. In other words, the participants have ‘arrived’ the target page that contains the 
necessary information to answer the question correctly but failed to choose the right answer. 

Based on the data from the task sheets, desktop screen recording, and direct observations, we knew that all 
participants efficiently “arrive” on the target page and successfully locate the necessary information. They also felt 
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confident about their search and answers. From a usability standpoint, it was a success. However, we do not know 
why some of participants get the second answer wrong when they have the information in front of them. It raises 
questions: what are the possible explanations and how can this be improved by design if applicable. The motivation 
to find answers has led to the present study. 


Purpose of the study 


The primary purpose of the present study was to explore possible explanations of why some participants 
respond with the “wrong” answer when they have “right” information in front of them by investigating user's 
information search process, including visual browsing/reading patterns, presentation style of the information used, 
and general performance (e.g. time spent, number of pages visited, completion rate etc.) 

The study was an extended study from the previous pilot study, using the same research design with a 
larger number of participant data set. Descriptive results regarding the search process and performance were 
reported. However, instead of focusing on how the users ‘arrive’ or ‘found’ information on the website, the question 
becomes how the users ‘read’ or ‘look at’ the information. 


Methods 


As mentioned earlier, the present study was a study extended from previous pilot study aiming to seek 
possible explanation with a larger data set. User research methods were adopted, which involved asking participants 
to perform a given task in a lab setting and directly observing what they do. A well-designed library website was 
applied to test out two search tasks commonly seen in libraries, which are ‘searching for library hours’ and 
‘eligibility of applying for a library card.’ All participants were performing these two tasks after the study was 
introduced and the consent of participation was obtained. Computer screen was recorded during the process. In 
addition, an eye-tracking device (Tobii 4C eye-tracker) was used to detect the visual pathway. 


Participant 


The target audience of the library website is general adults. Therefore, the general selection criteria were 
adults aged 20 years old or older with no severe visual impairments. Recruitment information was posted on 
Facebook and PTT, a social media tool used by students in Taiwan. Due to the geographic convenience, all 
participants were students from a public university where the lab was located. A total of 37 valid data sets were 
collected from the eligible participants. They are 27 females and 10 males with average age 26 years old. 


Data Analysis 


Study included both qualitative and quantitative data. In order to get an overall picture of searching and 
browsing process, descriptive data analysis was employed, describing demographic data, task completion, time 
spent, and the average number of web pages browsed. In addition, screen recording with scan path video was further 
used to analyze the user’s visual behavior patterns. Qualitative content analysis was applied to analyze videos, 
which inquired about visual patterns and type of message display. Video were reviewed and coded into emerged 
categories. The majority of respondents were under the age of 30, and 4 of the 37 people aged 30 or older. 


Results & Discussion 


The efficiency and effectiveness of online information searching performance were evaluated via various 
indicators, such as completion rate, time spent on tasks, number of pages/clicks count, accuracy etc. (Goldberg, 
Stimson, Lewensteln, Scott, & Wichansky, 2002). In this study, we reported completion rate, number of pages 
browsed, time spent on tasks, and accuracy of responses. To illustrate the qualitative differences as well as 
triangulate possible explanations, results also include the type of information display (e.g., table vs. paragraph) and 
visual patterns that emerged from the scan path video. 


Completion rate 


Same as defined in the pilot study, the definition of technical completion is to ‘arrive at the “target web 
page,” which refers to where the necessary information for completing tasks is located. The soft definition for 
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completion was when the participant indicated they were finished (writing down answers and return the task sheet to 
the facilitator). All participants completed the tasks. Overall the participants have no problem finding information in 
responding to the given tasks. All participants completed both tasks within four minutes. 


Browsed page count 


As mentioned earlier, the browse page count is one of indicators associated with search efficiency 
(Goldberg et al., 2002.) Results showed that all participants complete both tasks within eight pages, excluding the 
home page/starting page. Most of the participants completed the two tasks within two pages respectively. Twenty- 
nine out of 37 participants (78.38%) finished task one within two pages, and 32 out of 37 respondents (86.49%) 
completed task two within two pages. In other words, most of the participants completed each task with 2 clicks, 
which is considered rather efficient (see Table 1). 


Table 1. Browsed page count per task (n=37) 


Number of pages visited Task 1 Task 2 

N N 

1-page 20 24 

2-page 9 8 

3-page 4 3 

4-page 4 1 

7-page 0 1 

Total 37 37 

Time spent 


As shown on Table 2, the average time spent on task one and task two were 63.59 seconds, 51.26 seconds, 
respectively. The average time to complete task one and two was 114.85 seconds. We further split time spent into 
two parts: one was the time spent on searching (from the departure point to the arrival of the target page); the other 
was the time spent on the target page (where the necessary information was located). For task one, the average 
searching time was 33.14 seconds, and the average time spent on the result page (target page) was 30.45 seconds. 
For task two, the average time that participants spent on searching targeted page was 22.03 seconds, and the average 
time spent on the result page was 29.23 seconds. Comparing the time participants spent on the two different tasks, 
results showed spent longer time on the first task than those on the second task, both searching the target page and 
stayed on the target page. 


Table 2. Time spent on search, reading and completion 


Time (Seconds) Task 1 Task 2 
Average search time (starting point to target 33.14 22.29 
page) 
Average time stayed at the result page 30.45 28.80 
Average total time to complete task 63.59 51.26 
Accuracy 


All participants got the first question right and 11 out of the 37 respondents (29.7%) answered the second 
question incorrectly. The number of incorrect responses for the second question was “relatively” high while 
considering the ease of the task. During the observation process, all participants filled out answers confidently, and 
most importantly, they all had the right information in front of them. This aligns with the findings from the pilot 
study. The answer to the second question was “yes,” a high school student can apply for a library card. In general, 
only 16 years old or older can apply for a library card. However, full-time students under 16 years old can also apply 
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for a card. Both pieces of information were listed on the target pages. Unlike task one, which was straightforward, 
task two requires some thinking to process both ‘conditions. We know that participants have “arrived” the “target” 
pages that contain the necessary information for the question/inquiry and possible “look at” the information before 
answering it. However, we do not know if the participants actually “read” the information and how they read it. This 
is one of the areas where eye tracking data can provide additional information for better understanding without 
interrupting task process. We turned to desktop recording with eye tracking data for better insights and possible 
explanation. 


Display/layout of information 


How information is structured and display often plays an important role for reading (or browsing). The 
tasks were live search and participants were free to go to any page and use any strategies to find answers for the 
given inquires except using ‘search tool.’ Based on the data we collected, there were three “target pages’ where the 
participants “found” or “landed” for the inquiry #2 (task 2). One page presents the information in table format, 
whereas one was all text in paragraph format. The third page was a pdf document regarding library policy. The 
information was displayed in text form in paragraph as well. Therefore, we coded videos into two categories based 
on the format of display: table vs. paragraph. 

As shown in Table 3, 23 participants landed on table format and 14 participants landed on paragraph 
format. Among the participants in table group, 18 (78.26%) got the answer right while eight (57.14%) participants in 
paragraph group got the task 2 right. One possible explanation was the table format of information display helped 
participants read the information. 


Table 3. Task 2 responses by display format (n=37) 


Response Table Paragraph 
N % N % 
Correct 18 78.26 8 57.14 
Incorrect 5 21.74 6 42.86 
Total 23 100 14 100 


Moreover, for task 1, participants spent more time on ‘searching’ than ‘reading,’ whereas for task 2, 
participants took longer time on ‘reading’ than ‘searching.’ While reading time between the two tasks was not 
relatively meaningful, it is interesting to see the time difference on reading/finding information between two display 
styles (table vs. paragraph) regarding the task 2 responses. 

In general, participants who get the answer wrong spent more time on target page too. It could be that most 
participants who get the wrong answer landed on paragraph format, which naturally take more time to read and 
process information. As mentioned earlier, 23 out of 37 participants “landed” on the target page with table display 
and 14 on paragraph display. The average time for “table” group of participants to spend on the target page before 
writing down answer was 23.46 seconds while the average time spent on the target page for “paragraph” group 
38.72 seconds. This could mean that it took longer for participants to process information in the paragraph format 
than table format. This finding aligned with common design principles for organizing information. Table helps to 
visualize information with simplify form. The paragraph format also contains more text. However, this is not to 
suggest that all information needs to be organized in table format. It will depend on the context that is designed for, 
such as the content, the purpose, needs, and the audience etc. 


Table 4. Time spent on task 2 (n=37) 


Time (Seconds) Display format Response 
Table Paragraph Correct Incorrect 
(n=23) (n=14) (n=27) (n=11) 
Average search time 18.71 27.47 21.4 23.6 
Average time stayed on result page 23.46 38.72 26.5 35.7 
Average total time to complete task 42.17 66.19 35.7 59.3 
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Visual Pattern 

After reviewing all scan path video, two general patterns were identified. One was “focus,” which most of 
visual movement was more congregate or assembled. They also fall into the area of interest, the area that contain 
answers or necessary information to make a judgement (see Figure 1). The other type of visual pattern was “spread” 
or “scattered,” which by contrast, majority of visual attention fall outside of the area of interest or the visual path 
was (more) spread out (see Figure 2). 
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Figure I. Example of “focus” visual pattern 
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Figure 2. Example of “spread” visual pattern 


As shown in Table 5, the participants whose visual pattern was more focused or strategic have higher 
percentage getting the answer correct (96%) while the participants in spread or scattered visual pattern have a higher 
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percentage of people get the answer wrong. For a few cases, it was challenging to determine which category when 
there was a mix of patterns with high speed of visual movements. For example, the visual attention might focus 
(fixation) on one spot or one focused area but quickly move around for the rest of time and in some cases come back 
to the same spot. When it happened, the video was reviewed by the third person and discussed with all coders (total 
of three) until reach consensus. With that in mind, some cases are in the grey area. Another interpretation can be that 
the visual pattern of the participants who answered correctly was more ‘focused’ or strategic searching/browsing 
pattern, whereas the participants who did not respond with the correct answer showed no patterns or spread out 
pattern regarding the visual path when conducting the task. That could mean the participants 'skimming’ through a 
lot of text quickly without ‘reading’. 


Table 5. Task 2 responses by visual pattern (n=37) 


Response Focus Spread 
N % N % 
Correct (n=26) 23 96 3 23.07 
Incorrect (n=11) 1 4 10 76.93 
Total 24 100 13 100 


Among 23 participants who landed on the page with table format, 18 participants’ visual pattern was 
“focus.” And only one out of 18 got the wrong answer. While among 14 participants who landed on the paragraph 
format, there were only six participants’ visual pattern was “focus” and all six responded with correct answer (see 
Table 6). This finding could indicate that table format helps allocate visual attention and ultimate help the “reading.” 


Table 6. Task 2 responses correction, type of display used, and users’ scan path patterns (n=37) 


Type of display Scan path Task two 
N N N 
Used pattern response 
Table 23 Focus 18 Correct 17 
Incorrect 1 
Scatter 3) Correct 1 
Incorrect 4 
Paragraph 14 Focus 6 Correct 6 
Incorrect 0 
Spread 8 Correct 2 
Incorrect 6 
Total 37 37 37 
Conclusion 


With a larger data set (larger number of participants) and eye-tracking data, we were able to get more 
insights on how participants search/browsing the web content for inquiry-based tasks. We find that the findings 
regarding general search/browsing behaviors from the pilot still hold true. The website is well designed from a 
usability standpoint. All participants efficiently completed the tasks by “conventional definition,” which users 
arrived at the designated locations and spent approximately the same amount of time to search information and 
completed the tasks (whether they got the answer wrong). They have the same patterns to use both top navigation 
menu and the sitemap on the bottom of each page to get around the site. 

Eye-tracking data provide additional information that conventional methods cannot obtain, which helps to 
confirm or triangulate a specific phenomenon, in this case, their visual browsing/reading patterns on the target 
information. It took longer for participants to process information in paragraph format than table format, which is 
normal. However, more participants who landed on table format have the correct answer for task 2 than percentage 
in paragraph format. 

In general, we can conclude that table format works better than paragraph format for this type of 
information or inquiry. The table format in general guides visual attention to the target information if designed well. 
We may be able to design in a way to guide visual attention for better performance beyond just usability. However, 
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we cannot guarantee users to engage with the information they see even if they read it. It might require a different 
type of research for user engagement; specifically, the type of engagement with the content. More future research is 
needed. 

Based on the results, we can conclude that the characteristics of visual pattern for most participants in ‘right 
answer’ group was more focused while the pattern was more spread for the other group. We can conclude that for 
people in the right answer group seem to have more focused visual patterns while the other group have more spread 
or scatter visual patterns. In addition, the information in the table format seem to help users in ‘reading’ the 
information. 

Finally, individuals may have seen the information but not ‘read’ it carefully or the reading was not 
‘registered’ in the mind. Since it requires to put two conditions together for task 2, somehow the information that has 
been ‘seen’ or ‘read’ need to be engaged with the mind as well. However, we are not able to know for sure whether 
the information that has been ‘seen’ or even ‘read’ was ‘processed’. Further research is needed in exploring higher 
level information processing. 
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Abstract 


In response to evolving changes in the teaching and learning and in contrast to conventional screen 
recorded video tutorials with narratives, story-based videos were developed with cartoon characters, settings, and 
contexts similar to student life. Both types of videos are used for library instruction. A user study with 53 first-year 
students was conducted to examine student learning experiences with both types of videos. The present paper shares 
detailed results and comments from students, including their feedback on engagement, satisfaction, conference level, 
and video style preference. Results show that both types of instructional videos are equally effective in terms of 
immediately learning outcomes. However, students rated higher on engagement, satisfaction, confidence level on 
story-based/carton animation video and prefer story-based/cartoon animation if they were given an option. 


Keywords: action research; engagement; animation 
Introduction 


Online learning is a trend in changing learning environment and, in some cases (e.g., rural areas), deliver 
learning materials through the Internet is the only or the most efficient way to provide services for students at 
distances. As a result, it is important to design and create effective learning materials for online delivery. It is no 
exception to academic libraries in supporting student success through multiple modes and provide access to 
materials in variable formats (Arola, Sheppard, & Ball, 2014, Carlito, 2018, Cordes, 2009, Kress, 2010). Carlito 
(2018) suggested librarian may implement multimodal support to improve information literacy through a 
multimodal model, which consists of four strata, including discourse, design, production, and distribution. Video 
tutorials are one of mode that libraries use to deliver library instruction in helping students learn about library 
resources and how to effectively identify and utilize them for their academic needs. For example, Xiao, 
Pietraszewski, & Goodwin (2004) used online database instructional videos to provide contextual and relevant 
assistance in a just-in-time manner. Malone (2015) synchronous video were provided to online library instruction to 
respond the increased request for research consultations from online courses students. Librarians are familiar with 
multimodal deliveries of instructions and services. Carlito (2018) further suggested that the use of video clips, 
image, colors were the strategies to implement design element in library instruction. 

Like other teaching and learning professions, academic libraries follow best practices and several trends in 
the field of instructional design and scholarship of teaching and learning. This includes to utilize sounds 
instructional approaches (e.g., story-based learning) as well as effective educational technology (e.g. multimedia, 
animation, AR, VR, etc.) 

In education literature, successful experiments regarding the effectiveness of story-based learning have 
been reported (Kose, Koc, & Youcesoy, 2013; Kroth & Cranton, 2014; Shaw, Lind, & Ewashen, 2017; Wong & 
Poon, 2013; Wong et al., 2019). Story-based learning refers to the learning approach applied attractive stories on 
human learning process. In the field of education, stories have been considered as special and supportive elements to 
facilitate educational processes (Kose et al., 2013). A study of Shaw and colleagues (2017) found story-based 
learning enable nursing students enhance advanced communication skills in pediatric nursing. Moreover, researchers 
have indicated that stories possess a powerful to engaging the audience's emotions and affective, and the potential to 
transform their perspectives and actions (Kroth & Cranton, 2014; Wong & Poon, 2012; Wong et al., 2019). In short, 
story-based approach helps to make connection between the learners and the content by making the materials 
relevant. 

Another long proofing practice is the application of animation in education. Animation video clips can be 
effective instruction (especially young learners) because it: (1) provides students opportunities to involve in specific 
content interactively; (2) offers learns an opportunity to be engaged in their own learning process; and (3) allows 
students to share information through social media channels with friends around the world (Gurvitch & Lund, 2014). 
Yeh et al. (2012) suggested that applying animation appropriately is beneficial for reducing extraneous cognitive 
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load and improving students’ understanding of complex phenomena. Cook (2006) also suggested an animation- 
based approach is beneficial for students learning science. 

The value of engagement in student learning is no longer questioned (Trowler & Trowler, 2010). There is 
positive association between engagement level and learning outcome, the deeper engagement levels, the better 
learning outcome (Grissom, McNally, & Naps, 2003). In order to enhance the engagement and in response to 
evolving changes in the teaching and learning environment, our libraries re-designed several videos to promote a 
more engaging learning experience by incorporating new instructional strategies, such as micro-learning, story- 
based learning, and problem-based learning. A set of video-based library tutorials were created and designed to 
support distance education students and independent learners who want to learn the content themselves. This poster 
reports a practice of instructional design and development on a creation of library tutorial videos using story-based 
approach, specifically using cartoon characters and animations to present content that has been traditionally delivery 
through screencast with narrative. 

The study aims to collect information about learners’ perceptions of the different representations (e.g., 
video presentation style) with respect to student’s viewing experiences. This information can hopefully provide 
insight into the dialogues regarding the roles and influences of animation in instruction. The study was primarily 
driven by the following two questions: First, what is the impact of a story-based strategy on the user learning 
experiences when using online video tutorials (i.e., satisfaction, engagement, confidence, and preference). Second, 
what is the impact on the learning outcomes from such a story-based approach? 


Methods 


In order to achieve our research goals, we conducted a user study with first year students who are the 
primary or target audiences of the library video tutorials. We utilized methods from user experience research, which 
focuses on understanding users’ needs, behaviors, and motivations through task analysis, observation, and other 
feedback methodologies, including survey/questionnaire and contextual interviews. 

Two library video tutorials were used with the 53 first-year students to assess their experience in video 
watching and understanding of the video content. One video was a “conventional” show-and-tell video with a screen 
recording and voiced by a librarian. The other video was story-based with conversation by characters (i.e., 
conversations students have with the professor in the classroom). The contents of both videos were the same, which 
involved: (1) finding a specific database (in this case, PsycINFO) from the library home page, (2) conducting a 
keyword search for journal articles on given topics, and (3) refining a search to peer review journal articles. The 
length of the videos is also about the same. There is only a few seconds’ difference between them. 

Video format is the independent variable in the study, including study-based and non-story-based. Video 
watching experience, the dependent variable, was assessed after watching story-based video and non-story-based 
video, respectively. The investigator attempted to examine if college students’ watching experience in story-based 
tutorial video differ from non-story-based tutorial video. If one type of video was more preferable for participants, 
we would expect the score of watching experience is higher than the other type of video. 


Participants 


College students were recruited from a mid-size research university in the Midwest area in the U.S. Student 
is the main patron of the university libraries, as well as the main user of library tutorial video. Focusing the user 
experience on student allows librarian develop better understand user experience and create proper service 
experience for students. The current study further targeted first-year college student since they may have limited 
experiences in library tutorial trainings relatively compared with the students in their second- to fourth-year. In total, 
53 first-year colleague students were recruited for the study with ages ranging from 18-21 years old. 


Data collection procedures 


Ethical approval was obtained from the Institutional Review Board (IRB) of the author’s institution before 
the beginning for participant recruitment. Recruitment information was sent to all first-year students through email. 
A preliminary inquiry about participants eligibility was conducted when people contact the researcher indicated their 
participation interest. Written informed consent was obtained from each participant before participating in the study. 
Each participant participated in two sessions, one week apart. During the first session, all participants were asked to 
view videos, perform tasks (based on video content), and fill out a survey questionnaire regarding their video 
viewing experiences, including satisfaction, engagement, and preferences of video presentations. A quick follow-up 
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interview was followed to clarify questions raised during the session. Demographic data was also collected. At the 
second session, participants were asked to perform tasks based on the video content in previous session. Then watch 
the other video clip and fill out a survey regarding their video viewing experiences. A quick follow-up interview was 
followed to clarify questions raised during the session. 
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Figure 1. Data collection procedure 
Data collection tool 


A 22-item questionnaire was developed to assess participants’ video watching experience. Four questions 
related to demographic characteristics, such as gender, age, and academic status, and two questions assesses 
participants’ impression of video in general (i.e., preference and learning new). Sixteen Likert-scale questions (1= 
strongly disagree to 5= strongly agree) assess participants’ experience and understanding of videos from three 
aspects including engagement, satisfaction, and confidence. Questions about engagement assess how engaging the 
video is based on participant’s perspective. Questions related to satisfaction assess the feeling of pleasure regarding 
the videos. Finally, questions about confidence assess participants’ confidence level to perform the task that they 
just watched from the video. The reliability for engagement (a = .872), satisfaction (a =.792), and confidence (a 
= .821) scale were good respectively. The questionnaire was distributed in both sessions. However, the 
questionnaire in the second session did not include four demographic items. 


Data Analysis 


Descriptive analysis and pair-sample t-test were used for data analysis. The collected survey data was 
imported into SPSS. The descriptive analysis was used to present the features of the participants, and the pair- 
sample t-test was applied for data analysis to examine participants perception regarding the two types of video 
format. An alpha level of .05 was used for all statistical tests. 


Result & Discussion 


Demographic information 

As stated earlier, a total of 53 students participated in the study. The majority of the respondents were 
freshmen, came to school less than one year, and aged 18 to 21 years old. Thirty percent of the students were male, 
fifty-six percent of participants were female, and seven students did not answer. Descriptive statistics for 
participants demographic characteristics were listed in Table 1. 


Prior experience 

We wanted to know more about their prior experiences with library video tutorials and knowledge about 
the content covered in the video. So, the participants were asked if they have watched library tutorials before and if 
they learn anything new after viewing the video. Approximately 86% of the students reported they have never 
watched library tutorial videos before. Some of them have watched but they did not remember what it was at all. 
When asked, approximately 83% of the participants indicated they learned something new from the video; while 
about 10% of the students indicated that they did not. 
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Table I. Descriptive characteristics of the Survey Respondents (n=53) 


N % 

Academic Status 

Freshmen 50 94.3 

Sophomore 3 5.7 
Age 

18-21 years old 53 100.0 
Gender 

Male 16 30.2 

Female 30 56.6 

Not answered 7 13.2 


Engagement/ satisfaction/ confidence 

The results of pair-sample t-test for the mean value of engaging, satisfaction, and confidence after watching 
the story-based and nonstory-based library tutorial videos were listed in Table 2. Results showed the mean value of 
engaging rating, satisfaction, and confidence for story-based tutorial video is higher than nonstory-based video. It 
means participants had higher rating for story-based video than nonstory-based video. Results further indicated a 
significant difference in the mean of engaging rating between nonstory-based and story-based tutorial videos [t (51) 
= - 6.15, p < .001]. It means the participants felt more engaging in story-based tutorial video than nonstory-based 
tutorial video. On the other hand, the mean difference of satisfaction between story-based and nonstory-based video 
was not significant [t (52) = - 0.22, p = .831]. These findings reveal that participants’ level of satisfaction regarding 
story-based tutorial video and nonstory-based tutorial video are about the same. Similar results were found in 
participants’ level of confidence that the mean confidence between story-based and nonstory-based videos was not 
significant [t (52) = - 1.67, p =.102]. Such findings suggested that the participants’ understanding of video content 
stayed the same after watch story-based and nonstory-based videos. 


Table 2. Descriptive Statistics and t-test Results for Non-Story Based vs. Story-Based Video 


Nonstory based Story based 95% CI for 
Mean 
Outcome M SD M SD n Difference t df 
Engaging 3.70 0.62 4.17 0.52 52 -0.63, -0.32 -6.15* 51 
Satisfaction 4.26 0.83 4.29 0.79 53 -0.29, 0.24 -0.22 52 
Confidence 4.54 0.57 4.66 0.50 53 -0.27, 0.03 -1.67 52 


*p<.05. 


Video Preference 

When participants were asked about their preferences of presentation type (cartoon animation vs. 
convention screencast with narrative) if given an option, most participants (76.1%) choose cartoon presentation. The 
participants did not give any “deep” reasons. Most of them simply said because they like it. Some did provide a little 
explanation about cartoon style does not seem as heavy and ‘the other one was kind of boring.” On the other hand, 
some participants feel that carton video seems “a little bit cheesy.” 


Performance 

The unique aspect of this study was to investigate the video viewing experiences with multiple perspectives 
(1.e., engagement, satisfaction, confidence, and immediately learning outcome) and with actual target users, first- 
year students who have relatively few or no university library experience. As mentioned earlier, all participants were 
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able to complete the given tasks taught by the video within an average of three minutes. Both types of videos 
worked effectively on immediate learning outcomes (i.e., perform a similar task right after watching it). 


Conclusion 


Overall, students more engaged in story-based library tutorial video than nonstory-based tutorial video. 
Based on the results, we can conclude that the format of library tutorial video had an effect on college students’ 
engaging rating, but not satisfaction or confidence. In general, story-based video is more attractive for freshman 
student than nonstory-based video. If given a choice, they prefer story-based representation. The presentation style 
(or the representations) of the video did not make an effect on students’ understanding or impression regarding video 
content. In other words, both representations of video had no difference regarding the impact on immediate learning 
outcomes. While both types of video were equally effective, we should still try to make the viewing experience 
more engaging or even entertaining. However, the video clips used in the present study were less than three minutes. 
It would be worth exploring the effect of the video representations with longer lengths (or more content) on both 
immediate learning outcomes as well as retention (long term learning). 

This study provides insights for the instructional designer who helps with creating multimedia in support of 
distance education or eLearning as well as librarians who provide information literacy instruction; in particular, 
those using videos for online environment, and researchers who are interested in user experiences and instructional 
design. The author believes the story-based approach has great potential to enhance student learning in online 
settings; specifically, one-shot library instructions. This study tests the approach in an academic library setting with 
first-year college students. The results of the study potentially can lead to improvements in information literacy 
programs and inform library practices; in particular, those pertaining to designing, developing, and implementing 
better, more high-impact and innovative story-based approaches. 
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Introduction 


The literature reveals that courses on education technology are taught sporadically in teacher education 
programs and there are limited guidelines for course design (Hsu & Hargrave, 2000). Despite the increasing interest 
in integrating technology in classrooms, teachers often find themselves challenged by the lack of content-specific 
pedagogical strategies, and the time required to design classes (Bakir, 2016). In response to the need for baseline 
data for additional research and practice, the present study examines how education technology courses are taught in 
major teacher education programs in U.S. universities. 

By reviewing the literature, we found few studies or reports have been conducted to provide a national 
landscape view of how educational courses are taught in teacher education in the United States (U.S. Congress, 
Office of Technology Assessment [OTA], 1995; Hsu & Hargrave, 2000; Moursund & Bielefeldt, 1999). The OTA 
study (1995) has shown that upon graduation, preservice teachers learned little about technology and its application 
in teaching and learning. Hsu and Hargrave (2000) surveyed 88 teacher preparation institutions, and their study 
results showed a shift of course content from educational media or instructional design to computer technology in 
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teacher preparation. Their results also showed that compared with personal technology use or teacher productivity, 
there was a growing emphasis on integrating technology with curriculum. On the other hand, in recent years scholars 
and teacher educators have been advocating a stand-alone educational course in teacher education programs (Fedon, 
2018; Jia, Jung, & Ottenbreit-Leftwich, 2017). They argued that together with method courses and field experiences, 
preservice teachers will be better prepared to integrate technology in their teaching (Fedon, 2018). 

A major shift occurred in teacher education with the establishment of the International Society of 
Technology in Education standards (ISTE), previously known as the National Educational Technology Standards 
(NETS) (Bakir, 2016). In 2013, the National Council for Accreditation of Teacher Education (NCATE) adopted 
ISTE standards and required teacher education programs to restructure accordingly. The ISTE standards contributed 
to teacher education program by outlining and guiding fundamental concepts, skills, and attitudes pertaining to 
technology applications in educational settings (ISTE, 2008). However, there is a limited study on how ISTE 
standards are applied in the field. 

Extensive studies have been conducted on innovative ways to improve preservice teachers’ technology 
integration skills in teacher education programs. A study by Cherner and Curry (2017) showed that preservice 
teachers tend to use “low-level” technology in their practice. That is to say, instead of using technology to redefine 
or transform learning experiences for students, preservice teachers utilized technology to as a replacement or 
enhancement of existing educational activities. Results showed that the level of using technology shift higher as 
preservice teachers became more experienced and progressed through their internship. In another study, Zipke 
(2018) reported on the impact of a newly designed course that emphasized authentic learning experienced and field- 
based applications. Results showed an improvement of technology use in preservice teachers’ lesson planning. On 
the other hand, several studies have shown that preservice teachers improved proficiency in technology use and felt 
interested in utilizing technology after various interventions, but they were not confident to use technology in their 
own class later on (Fedon, 2018; Zipke, 2018). Zipke (2018) suggested that it was crucial for preservice teachers to 
have constant training, modeling, and support from inservice teachers who were tech-savvy in order to extend what 
they have learned in teacher education programs. By reviewing the literature, we found that more recent data is 
needed for deeper and more vigorous analysis on how educational technology is taught in teacher education. The 
following questions guided our research agenda: 

1. How is education technology taught in teacher education programs in the United States? 

What are the foci of learning objectives in educational technology courses? 
How are these course objectives aligned with ISTE Standards for Educators? 
What pedagogical features do education technology syllabi emphasize? 
What sources do educational courses use as text or reading(s)? 

What technological skills are embedded in educational technology courses? 


SP web 


Theoretical Framework 


Ample studies have been conducted to examine pedagogical features associated with positive changes in 
teachers’ knowledge, skills, attitudes, and teaching practices. For example, Darling-Hammond et al. (2017) 
identified seven course characteristics of effective professional development based on their extensive review of the 
literature. Courses for technology integration were also studied (An, 2018). By reviewing the literature (An & 
Reigeluth 2011; Asghar, Ellington, Rice, Johnson, & Prime, 2012; Garet, Porter, & Desimone, 2001), we have 
developed a combined codebook of pedagogical features (RQ3), which includes eight categories: subject specific, 
active learning, coherence, models and modeling, coach and expert support, collaboration, feedback and reflection, 
and technical skills. Description of each category is listed in Table 1. 


Table 1. Effective pedagogical features of professional development 


Subject focus This element includes an intentional focus on discipline-specific curriculum 
development and pedagogies in areas such as mathematics, science, or literacy. 
(Darling-Hammond et al., 2017; Asghar et al., 2012). 
If a course has assignments or other course elements that entail subject-specific 
technology integration ideas, it would also count (An & Reigeluth, 2011). 


Active learning Active learning engages pre-service teachers directly in designing and trying out 
teaching strategies, providing them an opportunity to engage in the same style of 
learning they are designing for their students. Such PD uses authentic artifacts, 
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interactive activities, and other strategies to provide deeply embedded, highly 
contextualized professional learning. 


This approach moves away from traditional learning models and environments that are 
lecture based and have no direct connection to classrooms and students. (Darling- 
Hammond et al., 2017). 


Coherence A PD activity is more likely to be to effective if it forms a coherent part of a wider set 
of opportunities for pre-service teacher learning and development. (Garet et al., 2001; 
Asghar et al., 2012). 
Three dimensions: 
the extent to which it builds on what teachers have already learned; 
emphasizes content and pedagogy aligned with national, state and local standards, 
frameworks, and assessments; and, 
support teachers in developing sustained, ongoing professional communication with 
other teachers who are trying to change with their teaching in similar ways. 


Collaboration “Collaboration” can span a host of configurations—from one-on-one or small-group 
interactions to schoolwide collaboration to exchanges with other professionals beyond 
the school (Darling-Hammond et al., 2017). 


Use of Models and PD that utilizes models of effective practice has proven successful at promoting pre- 
Modeling service teacher learning and supporting student achievement. Curricular and 
instructional models and modeling of instruction help pre-service teachers to have a 
vision of practice on which to anchor their own learning and growth. The various kinds 
of modeling can include (Darling-Hammond et al., 2017): 
video or written cases of teaching, 
demonstration lessons, 
unit or lesson plans, 
observations of peers 
curriculum materials including sample assessments and student work samples 
concrete sample activities and modules that were ready to go (Asghar et al., 2012) 


Coaching and expert Coaching and expert support involve the sharing of expertise about content and 
support (who delivered evidence-based practices, focused directly on pre-service teachers’ individual needs. 
PD or provided (Darling-Hammond et al., 2017). 

support) 


Feedback and reflection Feedback and reflection are two other powerful tools found in effective PD; they are 
often employed during mentoring and coaching but are not limited to these spaces 
(Darling-Hammond et al., 2017). 
Feedback from peers 
Self-reflection of learning and practice (e.g. journal) 


Technological skills Technology skills in the context of designing learner-centered learning activities in 
their subject areas (An & Reigeluth, 2011). 


ISTE standards contribute to teacher education by outlining and guiding fundamental concepts, skills, and 
attitudes pertaining to technology applications in educational settings. In the current study, ISTE Standards for 
Educators (See Table 2) are used as guidelines to analyze learning objectives in the course syllabi that we collected. 
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Table 2. International Society of Technology in Education Standards for Educators (ISTE, 2016) 


Domain Description 


Learner Educators continually improve their practice by learning from and with others and 
exploring proven and promising practices that leverage technology to improve 
student learning. 


Leader Educators seek out opportunities for leadership to support student empowerment 
and success and to improve teaching and learning. 

Citizen Educators inspire students to positively contribute to and responsibly participate in 
the digital world. 

Collaborator Educators dedicate time to collaborate with both colleagues and students to 
improve practice, discover and share resources and ideas, and solve problems. 

Designer Educators design authentic, learner-driven activities and environments that 
recognize and accommodate learner variability. 

Facilitator Educators facilitate learning with technology to support student achievement of the 
2016 ISTE Standards for Students. 

Analyst Educators understand and use data to drive their instruction and support students in 


achieving their learning goals. 


Method 


Education technology courses for prospective teachers often go by different names, such as “Education 
Technology,” “Technology Integration,” and “Computer Applications in Teaching.” Most of these courses are 
offered by Schools/College of Education (SCOE) in the fall and spring semesters. 

In order to answer how education technology is taught in teacher education programs in the United States 
(RQ1), we selected one university with the largest preservice teacher enrollment in its state in 2017, according to 
National Teacher Preparation Data (US Department of Education, 2017), from each U.S. state as a representative 
university (total n=50). We then collected teacher education program information on these 50 universities’ websites 
and whether educational technology courses were included in their teacher education degree plan. Data is reported 
descriptively. 

To look into course objectives, course alignment with ISTE Standards (RQ2), and pedagogical features of 
educational technology courses (RQ3), we solicited course syllabi from the same education programs in RQ] via 
email. Contact information of course instructors was collected on university websites. If no instructor was found, we 
contacted the director of teacher education programs for a referral to relevant course instructors. Reminders were 
sent two weeks after the initial request. As the time of writing this conference proceeding, we have collected and 
analyzed 20 course syllabi from 12 universities in 12 states, as it turned out some universities have more than one 
educational course in teacher education. The following information was recorded for each syllabus (Merced, 
Stutman, & Mann, 2018): learning objectives, course assignments, required textbooks, and types of supplementary 
resources (e.g., journal articles, educational news, videos). 

The guidelines of Strauss and Corbin (1990) were followed in the coding of learning objectives (RQ2). 
First, each learning objective was independently coded by two researchers to generate a list of initial codes. Each 
code represented one single idea and if a learning objective had more than one idea, each idea counted once, which 
lead to a list of 187 learning objective ideas for a total 20 of course syllabi. Then the third and fourth researchers 
reviewed and verified the codes and resolved any disagreement. If there were any questions, the entire research team 
met again and revisited the raw data, codes, and made necessary adjustments to ensure data validity and reliability. 
Using the constant comparative method (Creswell & Creswell, 2017), we made coding modification, realignment, 
and refinement during the coding and recoding process. This process went through iterative cycles until codes were 
categorized, and 100% inter-rater reliability was reached. After we categorized learning objectives, we coded it with 
ISTE Standards for Educators (See Table 2) to see to what extent they were matched. 

Course pedagogical features (RQ3) were analyzed qualitatively (Creswell & Poth, 2013). We first looked 
for instances in course syllabi that reflected pedagogical features in the Effective Pedagogical Features of 
Professional Development framework (See Table 1). Each syllabus was coded by at least two researchers and 
researchers met regularly to compare notes. Discrepancies were discussed and resolved until 100% inter-rater 
reliability was reached. 
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In the following section, we present the results of our preliminary study, an analysis of 20 course syllabi 
from 12 U.S. universities. Course short names! would be used to denote courses when we present data, such as ISTC 
302 or EDTL 6272. 


Results 


Educational Technology Courses in The United States. According to our results, 32 out of 50 
universities (64%) require an education technology course(s) for at least one certification area in their teacher 
education program. Among the 32 universities, 18 universities (36%) offer education technology courses in all their 
teacher education certification areas. Some universities only require students from certain majors to take the course 
(n=10, 20%). For instance, an education technology course is only required for undergraduates in Mathematics and 
Science teacher certification at the University of Georgia. 

There are 16 out of 50 universities (32%) do not provide education technology course in undergraduate 
teacher education programs. Instead, educational technology is embedded in other courses taught in teacher 
education. Among the 16 non-edtech-course universities, there are three universities (6%) that require the course in 
their graduate level teacher education programs. In addition, two universities (4%) have an undergraduate minor in 
educational technology, which is an add-on to their preservice teacher programs, and aim for preparing teacher 
candidates for technology integration within the instructional contexts where they will teach. 

For the 20 course syllabi we collected so far, four of them are optional courses, the other 16 courses are 
required for at least one certification area in teacher education. Most courses are for undergraduate level (n=17); 
three course delivery formats were revealed: face-to-face (n=6), blended (n=8), and online (n=6). 


Foci of Learning Objectives. Among 187 learning objectives from 20 course syllabi, a total of seven 
categories emerged (See Table 3). Results show that designing and developing a technology-rich environment to 
facilitate K-12 student learning is the most frequently mentioned learning objective (42.2%; n=79), which usually 
involve master technical skills and incorporate digital tools with curriculum and instructional materials. Promoting 
teacher candidates’ digital literacy is another major focus of these courses (18.2%; n=34), which include evaluating 
and selecting new information resources, implementing digital assessments and using resulting date to inform 
teaching. Developing knowledge around ethical issues of technology integration is mentioned less often (9.1%; 
n=17) and focused on including digital equity and responsible social interactions. A certain amount of learning 
objectives has shown a focus on teachers’ awareness of developing digital literacy and 21“ century skills of K-12 
students (8.0%; n=15), such as critical thinking, creativity, problem-solving skills and collaboration. The category of 
developing professional learning networks, on the other hand, is less covered in these courses (11.2%; n=21). 
Teachers are taught to continue to learn and develop their skills within and outside of an digital learning community. 
In terms of demonstrating pedagogical knowledge and skills (7.0%; n=13), 13 learning objectives out of 187 are 
categorized in this category. Even though a majority of course objectives are aligned with certain national standards, 
only in a few courses student teachers themselves are taught to know or use these standards. Only 8 learning 
objectives among 187 mentioned student teachers should properly explain standards to K-12 students and their 
colleagues (4.3%; n=8). 


Table 3. Learning objectives in educational technology courses 


Categories Example Percentage 
A Demonstrate knowledge, skills and Design and customize technology-enriched 42.2% (n=79) 
participation of designing and learning experiences to engage students in 
developing technology-enriched activities that deepen understanding in core 
learning experiences. subject areas (ED 308/CE 444). 


B_ Demonstrate proper attitudes towards Model and facilitate effective use of current 18.2% (n=34) 
technology; develop digital literacy to and emerging digital tools to locate, analyze, 
support teaching and learning. evaluate, and use information resources to 
support research and learning (ED 408). 


' Course short names were modified to protect data privacy. 
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C Demonstrate concerns and Demonstrate the basic principles of 9.1% (n=17) 


consideration of ethical issues in computer ethics and legalities to ensure 
technology integration and use, compliance by professionals and students 
including digital etiquette and with laws, guidelines, licenses and security 
responsible social interactions. in the use of all media (ED 305). 
D_ Knowledge of K-12 student learning in Identify and discuss the critical skills, 8.0% (n=15) 


a digital age; focus on using various referred to as 21st century skills, essential to 
digital tools to promote students 21st student success and vocational preparation 


century skills. in the 21st century and beyond (ED 307). 
E Develop professional learning network, Craft a professional learning network, 11.2% (n=21) 
continue to learn and develop their articulating clear goals for continued 
skills within and outside of an professional growth and documenting the 
electronic learning community. development of digital skills (EDTL 6270). 


F Demonstrate pedagogical knowledge | Compare and contrast a variety of partnering 7.0% (n=13) 
and skills. pedagogies, including inquiry-, problem- 
and project-based methodologies (ED 307). 


G_ Knowledge of standards and Explain the use of ISTE NETS for students 4.3% (n=8) 
requirements. and teachers (ED 410). 
Total 100% (n=187) 


Course Alignment with ISTE Standards for Educators. Our results show an uneven distribution of 
course content aligning with the ISTE Standards for Educators (2016) (See Figure 1). Most of course learning 
objectives fell into Designer category (n=92, 47%), that is to design technology-enriched learning experience for 
students. However, compared with ISTE’s emphasis on designing authentic and learner-driven activities (ISTE, 
2016), most learning objectives in our data corpus put more emphasis on the mastery of computer skills and general 
technology integration in curriculum. For example, /earn how to use a variety of technology and media resources 
(ISTC 301), develop the basic understanding of productivity and utility software capabilities and be able to use a 
variety of applications (EDLT) and gain experience in planning to integrate technology into the classroom 
curriculum (ISTC 301). 

There are 42 learning objectives (21%) that could be categorized in Citizen domain of ISTE Standards 
(ISTE, 2016). To be specific, most of the content (n=32, 17%) covers the digital literacy aspect of Citizen, e.g. the 
“establishment of a learning culture that promotes curiosity and critical examination of online resources and fosters 
digital literacy and media fluency” (ISTE, 2016, p.1) while only a portion (n=10, 5%) covers the ethical aspects of 
Citizen, e.g., “Mentor students in the safe, legal and ethical practices with digital tools and the protection of 
intellectual rights and property” (ISTE, 2016, p.1). 

Following Citizen, the next ISTE categories that have the largest number of learning objectives in our data 
corpus is Learner (n=16, 8%) and Facilitator (n=16, 8%). First, to be a successful educator in the digital age, ISTE 
requires teachers to continually improve their knowledge and practice of utilizing technology in a professional 
learning network. On the other hand, ISTE stressed that teachers need to utilize technology to facilitate student 
learning in digital age, e.g. taking ownership of their learning goals and outcome, becoming digital content creator. 
However, our data shows more emphasis on facilitating student critical thinking, creativity and communicative 
skills. For instance, promote the development of 21st century skills such as creativity, critical thinking, problem 
solving, and communication (ED 308). 

As for Collaborator (n=14, 7%), our data shows that most of the collaboration in educational technology 
courses refers to collaboration with colleagues, while ISTE Standards have a unique aspect of collaborating and 
communicating with K-12 students (ISTE 2016). Given the context of these courses are taught in preservice teacher 
education programs, it is understandable that course instructors do not put much focus on collaboration with K-12 
students. 
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Results show a limitation of teaching pre-service teachers to be Leaders (n=7, 4%) or Analysts (n=9, 5%) in 
educational technology courses. According to ISTE’s description on Leader (2016), educators need to seek out 
opportunism to engage with educational stakeholders for student learning empowerment, model digital literacy for 
colleagues, and advocate for digital equity for all students. Our data show all 7 learning objectives are modeling 
effective technology use for colleagues and K-12 students. As for Analyst, there are nine learning objectives that 
cover using digital assessment tools and having data-driven decisions. 


Analyst, 9, 5% Learner, 16, 8% 
Facilitator, 16, 8% 


Leader, 7, 4% 


= Learner 


Citizen, 42, 21% = Leader 
= Citizen 
= Collaborator 
= Designer 
= Facilitator 


= Analyst 


Designer, 92, 47% Collaborator, 14, 7% 


Figure 1. Course Alignment with ISTE Standards 


Pedagogical Features. Results show an extremely uneven distribution of pedagogical features among the 
20 courses we examined. First, 19 courses were not subject-specific, except for EDTL 6270 Technology and the 
Reading Classroom. There were 14 courses had active learning opportunities that engaged students directly in their 
practice. For example, in ED 305 Technology & Media in Education, student teachers are required to use SoftChalk 
course authoring software to create a developmentally appropriate lesson covering the topic of digital citizenship. In 
this assignment, students are given a chance to develop and practice skills that they will use for their future 
classrooms. 

More than half of the courses required group projects that entailed collaboration. Four courses incorporated 
models of effective practice in their instruction, either from inservice teachers or previous students. There are 19 
courses that required students to master some amount of technical skills. Regarding Coherence, 18 courses touched 
on at least one dimension, but most of them were not comprehensive. For instance, 18 out of 20 courses are aligned 
with national or state standards in their syllabus (Coherence 2), but only two courses mentioned that the course 
content is built on what teachers have learned before (Coherence 1) and seven courses had provided learning 
chances that support teachers in developing sustained, ongoing professional communication. For Coaching and 
expert support, four courses indicated that students should talk to the course instructors and receive guidance prior to 
some assignments. Over half of the courses used peer feedback and provided opportunities for reflection. 
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Figure 2. Distribution of course pedagogical features 


Readings and Resources. Among 20 courses, six courses required textbooks in their syllabi and only one 
course required students have a lecture capture software, SnagIt. The other courses used a variety of resources, 
including news reports, materials written by instructors, blogs, educational websites, videos, and journal articles. 
Four courses provided supplementary resources for student learning, which included educational websites, blogs, 
news websites, videos, government report, state standards, journal articles, and book chapters. Notably, the types 
and range of instructional resources varied drastically from course to course. For example, one course does not 
require a textbook, but instead offers a wide range of resources from practitioner perspectives, including news, 
EdTech news, and government reports. On the other hand, another course used only one type of resource (journal 
articles) for its required and supplementary materials (See Appendix I). 


Technological Skills. In the examination of educational technology course syllabi, we identified four 
categories of technological tools that preservice teachers were required to learn in class, including classroom 
hardware, productivity software, Web 2.0 technology and instructional software. We used the following inclusion 
criteria: 

Classroom hardware — tools and equipment that teachers and/or students use in class. 

Productivity software — application software used for producing information, such as documents, presentations, 
worksheets, charts, graphs, and digital video. 

Web 2.0 technology — tools and websites that emphasize user-generated content and participatory culture for end 
users. 

Instructional software — application software that include text, pictures, sounds, animations, and other various media 
used specifically to teach content-specific knowledge and skills. 


Our results show that most technological skills that were required from preservice teachers were the 
mastery of productivity software (n=82, 58%). It reflects that producing digital content, e.g., slides, digital lesson 
plan and images, is seen as the most important technological aspect in teacher education. Another relatively large 
aspect is the use of Web 2.0 tools (n=39, 28%). What is worth mentioning is, besides the fluency in using Google 
Suites, many courses require preservice teachers to create websites or blogs as course assignments. As for 
instructional software (n=14, 10%), most of applications mentioned in technology courses for pre-service teachers 
are learning manage system (LMS), while it also includes lecture capture software (SnagIt), and online assessment 
tool (Quizlet). Classroom hardware is the least mention technology tools for preservice teachers to master (n=6, 
4%), which include assistive technology, laptop, interactive white boards (IWB) and Clickers, an assessment tool. 
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= Web 2.0 technology = Instructional software 


Discussion & Implications 


According to the results from our preliminary study, we found remarkable variations in the ways education 
technology is taught in teacher education programs. Each course had its own emphasis either on technical skills, 
pedagogical knowledge, or history and theories, reflected in the resources they cited/utilized. The present study 
could provide meaningful information for designing and teaching educational technology in teacher education 
programs while providing a current snapshot of the field. 

Advocate for a stand-alone technology course in teacher education programs. First, we found out that 

only part of teacher education programs in our sample offered educational technology courses. Previous 
studies have suggested that a stand-alone technology course is beneficial for teachers’ knowledge and skill 
development, and future practice with technology (Fedon, 2018). Thus, we suggest there should be a place for 
educational technology courses in teacher preparation sequence. 

More content specific educational technology course in teacher education is needed. Among all 20 
courses we examined in the study, only one is subject specific. However, literature has shown that one major feature 
of effective professional development is discipline specific (Darling-Hammond et al., 2017; Asghar et al., 2012; An 
& Reigeluth, 2011). They argued that this type of professional development gave teachers “the opportunity to study 
student work, test out new curriculum with their students, or study a particular element of pedagogy or student 
learning in the content area” (Darling-Hammond et al., 2017, p.5). Technology courses should have content specific 
focus as well (An, 2018). 

Further effort is needed to align course content with the national technology standards. Our data 
shows that even though some courses were designed in line with some aspects of national standards, there are some 
improvement needed. First, instructors should update their references with a newer version of the standards. For 
example, we found out the prevalent use of NETS in course syllabi while NETS has changed its name and content to 
ISTE in 2007 and has released a couple of updated versions since. Second, a more comprehensive and in-depth 
reflection of ISTE standards needs to be considered at designing educational courses. Our results show an uneven 
distribution of alignment with ISTE standards. Nearly half of the courses learning objectives focus on preparing 
preservice teachers to become Designer (e.g., designing technology-enriched learning experiences for K-12 
students), while only 4% has touched on developing professional learning network (e.g., Learner). What’s more, 
even within the most popular Designer category, many courses didn’t reflect the essence of ISTE standards. 
Compared with ISTE’s emphasis on designing authentic and learner-driven activities (ISTE, 2016), most learning 
objectives in our data corpus remains on the level of the mastery of computer skills and general technology 
integration in curriculum. 

Technology in courses. Our study suggests that technological skills embedded in educational technology 
courses should move to a higher-level, e.g., from productivity to instructional software, from teacher-centered to 
student-centered. Our study has confirmed previous study results that the focus of technology use in teacher 
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preparation is still teacher centered and often involves productivity and efficiency (Cherner & Curry, 2017; Hughes, 
Cheah, Shi, & Hsiao, 2019). 


Limitation 


The current study has a few a limitation. First, it collected data from only 50 universities that prepare the 
largest number of teachers. Second, for RQ2 and RQ3, we collected and analyzed course syllabi, which are limiting 
in many ways. Course syllabi outline the basics of a course, but they do not describe how the course is actually 
implemented in practice. Some course syllabi were less detailed than others and didn’t include enough description of 
assignments or tasks. Despite these limitations, the need for examining how educational technology courses are 
taught is needed in the field. 


Conclusion 


This qualitative research study collected data on teacher education programs and examined course syllabi 
of education technology that are taught in major teacher education programs in U.S. universities. We are interested 
in how the course is taught and what pedagogical features do these courses emphasize. Our results revealed a wide 
range of course objectives, which mostly focus on developing student teachers’ knowledge and skills to design and 
develop a technology-rich environment; on the other hand, ethical issues and professional learning are covered less. 
Most courses are non-subject specific but certain assignments would involve students to incorporate subject ideas. 
The present study could provide meaningful information for designing and teaching educational technology in 
teacher education programs while providing a current snapshot of the field. 
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Appendix I Required & Supplementary Resources in Educational Technology Courses: 


Required/Optional 
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Optional 


Optional 


Optional 


Optional 


Optional 


Optional 


Optional 


Types 
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Book 


Book 
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Book 
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Book 


Book 


Book 


Book 


Book 


Book 


Book 


Book 


Titles 
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Roblyer, M. D. and Hughes, J. E. (2019). Integrating Educational Technology 
into Teaching (8th ed.). New York, NY: Pearson Education, Inc. 
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Corwin 


Bender, W. B. (2012). Technology and the New Differentiated Instruction 
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Corwin 
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Abstract 


The goal of this paper has two folds: First was to analyze the research articles published in 3 widely accepted 
journals published in Turkey to be able to explore the current trends in the field of ODL research during the period 
2010-2015. Based on this analysis, second, it it tries to draw a big picture about the issues and problems in ODL 
research in Turkey. The results, or lessons drawn from this study can be summarized as: technology integration to 
traditional education is gaining importance, the focus of the research studies has shifted from teachers to learners, 
educational technology, learner characteristics as well as theories and models are the most common research areas, 
achievement is the most focused variable among Turkish researchers, quantitative research method is also the most 
popular research method, the references and authors cited in the articles are quite diverse from all over the world. 


Introduction 


Turkey met open and distance learning nearly sixty years ago and then due to the advancements in information 
and communication technologies (ICT), open and distance learning has evolved deeply. In addition to these 
developments, new implementations, pedagogies, theories and models have occurred globally in the field of open and 
distance learning. Harasim (2000) states that Web technologies made online education increasingly open, accessible 
and flexible which allowed new pedagogical models to emerge and reasoned the revolution in digital knowledge age 
that enabled greater and faster human communication and collaboration and led to fundamentally new forms of 
economic activity that produced the knowledge economy and required basic changes in education. 

Turkey has always been trying to keep up with these improvements. Today, the number of distance education 
providers in Turkey has reached to 88. According to Higher Education Council (HEC) statistics (2018), there are 
nearly 8 million students in higher education and nearly 2 million of them are distance learners. This means that there 
is great demand for open and distance learning. In order to be able to response this demand, educators and 
administrators must have the latest knowledge and skills about current trends in the field. 

Although open and distance learning has a long tradition in Turkey in terms of practice, the research published 
in Turkish journals is a bit less than it has to be. In order to spot the open and distance learning trends and provide a 
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detailed list of current research trends in Turkey, this study proposes to fill the gap in the field of open and distance 
learning research conducted and published in Turkey. 


Literature Review 


Systematic review of research has been accepted as a research method for some time as a result of the studies 
employed this method over the last couple decades. In the field of ODL, we observed several major systematic reviews. 
One of them, for instance, was carried out by Berge and Mrozowski (2001) who examined the ODL literature between 
the years 1990 and 1999. In this research, the author reviewed 890 articles under the categories proposed by Sherry 
(1996): participants, technology selection and adoption, design issues, strategies to increase interactivity and active 
learning, learner characteristics, learner support, operational issues, policy and management issues, equity and 
accessibility, and cost/benefit trade-offs. The results showed that pedagogical themes and strategies for active learning 
were the most frequent topics used in the field. 

Another major study was conducted by Lee, Driscoll and Nelson in 2004. They examined 383 articles 
published in four journals (The American Journal of Distance Education-AJDE, the Journal of Distance Education - 
JDE; Distance Education-DE, and Open Learning-OL) from 1997 to 2002. The authors have examined the articles 
based on topics, research method, the statistical analyses, citation of authors, and the cited books. 

Zawacki-Richter, Backer, and Vogt (2009) conducted an extensive study by examining the 695 articles 
published in five prominent journals of ODL. The authors limited their study with the articles published between the 
years 2000 and 2008. Their study concluded that interaction and communication patterns in computer-mediated 
communication, instructional design issues, learner characteristics, and educational technology are the most frequently 
investigated topics in ODL research domain. Later, in one another study, Zawacki-Rihter (2009) developed a set of 
categories showing the research areas in the field of OLD which became one of the widely used categorization of the 
research topics in ODL. To be able to make this categorization, a Delphi study and an extensive literature review were 
carried out. As a conclusion, 15 research areas that were organized into 3 broad categories were created (Table 1). 


Table 1. Classification of Research Areas in Distance Education 
Macro level: Distance education systems and theories. 
1. Access, equity, and ethics: The democratization of access to DE afforded by new media and by finding 
ways to deliver high-quality education to those who have limited resources and poor infrastructure; issues 
that refer to the (sustainable) provision of DE in developing areas. What is the impact of DE (e.g., via mobile 
learning) on narrowing the digital divide and what is the role of ICT (information and communication 
technologies) and/or OER (open educational resources) in terms of access to education? 
2. Globalization of education and cross-cultural aspects: Aspects that refer to the global external 
environment and drivers, the development of the global DE market, teaching and learning in mediated global 
environments, and the implications for professional development. 
3. Distance teaching systems and institutions: DE delivery systems, the role of institutional partnerships 
in developing transnational programmes, and the impact of ICT on the convergence of conventional 
education and DE institutions (hybrid or mixed mode). 
4. Theories and models: Theoretical frameworks for and foundations of DE, e.g., the theoretical basis of 
instructional models, knowledge construction, interaction between learners, or the impact of social 
constructivism learning theories on DE practice. 
5. Research methods in distance education and knowledge transfer: Methodological considerations, the 
impact of DE research and writing on practice, and the role of professional associations in improving 
practice. Literature reviews and works on the history of DE are also subsumed within this area. 
Meso level: Management, organization, and technology. 
6. Management and organization: Strategies, administration, and organizational infrastructures and 
frameworks for the development, implementation, and sustainable delivery of DE programmes. What is 
required for successful leadership in DE? DE and policies relating to continuing education, lifelong learning, 
and the impact of online learning on institutional policies, as well as legal issues (copyright and intellectual 
property). 
7. Costs and benefits: Aspects that refer to financial management, costing, pricing, and business models in 
DE. Efficiency: What is the return on investment or impact of DE programmes? What is the impact of ICT 
on the costing models and the scalability of DE delivery? How can cost effective but meaningful learner 


support be provided? 
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8. Educational technology: New trends in educational technology for DE (e.g., Web 2.0 applications or 
mobile learning) and the benefits and challenges of using OERs, media selection (e.g., synchronous vs. 
asynchronous media), technical infrastructure and equipment for online learning environments, and their 
opportunities for teaching and learning. 

9. Innovation and change: Issues that refer to educational innovation with new media and measures to 
support and facilitate change in institutions (e.g., incentive systems for faculty, aspects referring to staff 
workloads, promotion, and tenure). 

10. Professional development and faculty support: Professional development and faculty support 
services as a prerequisite for innovation and change. What are the competencies of online teachers and how 
can they be developed? 

11. Learner support services: The infrastructure for and organization of learner support systems (from 
information and counselling for prospective students about library services and technical support to career 
services and alumni networks). 

12. Quality assurance: Issues that refer to accreditation and quality standards in DE. The impact of quality 
assurance and high quality learner support on enrolments and dropout/ retention, as well as reputation and 
acceptance of DE as a valid form of educational provision. 

Micro level: Teaching and learning in distance education. 

13. Instructional design: Issues that refer to the stages of the instructional design process for curriculum 
and course development. Special emphasis is placed on pedagogical approaches for tutoring online 
(scaffolding), the design of (culturally appropriate) study material, opportunities provided by new 
developments in educational technology for teaching and learning (e.g. Web 2.0 applications and mobile 
devices), as well as assessment practices in DE. 

14. Interaction and communication in learning communities: Closely related to instructional design 
considerations is course design that fosters (online) articulation, interaction, reflection, and collaboration 
throughout the learning and teaching process. Special areas include the development of online communities, 
gender differences, and cross-cultural aspects in online communication. 

15. Learner characteristics: The aims and goals of adult learners, the socioeconomic Background of DE 
students, their different learning styles, critical thinking dispositions, and special needs. How do students 
learn online (learner behavior patterns, learning styles) and what competencies are needed for distance 
learning (e.g., digital literacy)? 


One of the major systematic reviews carried out by Turkish researchers (Bozkurt et al, 2015) included 861 
articles published during 2009 and 2014 in the 7 peer-reviewed and indexed by the major well-known indexes: The 
American Journal of Distance Education (AJDE), Distance Education (DE), The European Journal of Open, Distance 
and e-Learning (EURODL), The Journal of Distance Education (JDE), The Journal of Online Learning and 
Technology (JOLT), Open Learning: The Journal of Open, Distance and e-Learning (OL) and The International 
Review of Research in Open and Distributed Learning (IRRODL). The authors also conducted content analysis and 
social network analysis. Their main goal was to see if there were any changes in terms of trends and issues in ODL 
research after the Zawacki-Richter’s study (2009). Their study concluded that distance education and open and 
distance learning terms are being used more frequently than before, which is considered as a paradigm shift in 
education. 


Purpose and Research Questions 


Although it is possible to find studies showing trends and issues in DE field on international basis from 1990s 
to 2014, there are no studies showing the trends of Turkish research domain published in a Turkish journal. It is also 
possible to find some similar studies showing the current trends about education technology (Géktas et al., 2012, 
Bozkaya et al., 2012) but none of them has a specific focus on open and distance education. This study was carried 
out to fill the gap and also to show the research areas most frequently used in the Turkish research domain. In addition, 
this study aims to provide comprehensive information for further research and to set light to recent ODL trends that 
might be useful for researchers. 

To be able to provide comprehensive information and to brighten the research domain following research 
questions were used: 

What are the most frequent/ly 

1. indicated keywords, 

2. chosen research areas, 
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emphasized theoretical/conceptual backgrounds, 

employed research designs, 

used data collection instruments and data analysis techniques, 
focused variables, 

targeted population and/or participant groups, 

cited references, 

cited authors in ODL research articles between 2010 and 2015? 


Seer SEONG Ee 


Method 


This descriptive study was conducted with the same methodology as the study by the Bozkurt et al. (2015) 
due to the fact that both were conducted as a part of a doctoral level course offered by the same professor, Cengiz 
Hakan Aydin. So, a content analysis was employed to analyze the articles for the purpose of answering the questions 
identified. Berelson (1952) defined content analysis as a systematic and replicable technique for classifying many 
words of text into fewer categories based on explicit rules of coding. After choosing the research design, a set of 
criteria was developed for the themes to be studied and the content of articles in the selected journals was coded 
according to these pre-set categories. Journals that are published by a Turkish institution in Turkish, a publication 
history of 10 years or more, a specific focus on DE and open and distance learning context, and indexed by the web 
of science indexes included into the study. Total four journals met the criteria: Hacettepe University’s College of 
Education Journal (Hacettepe Egitim Faktiltesi Dergisi-HEF), Theory and Practice of Educational Sciences (Kuram 
ve Uygulamada Egitim Bilimleri Dergisi-KUYEB), Journal of Education and Science (Egitim ve Bilim Dergisi), and 
Tiirkiye ve Orta DoSu Amme Idaresi Enstitiisii Dergisi (TODAIE). Two of these journals (KUYEB and TODAI) were 
excluded the web of science’s indexed after this study. Also, during analyses, no articles directly related to the field 
of ODL was found in one of these four journals, namely in TODAIE. So, it was excluded from the study. 


Table 2. Journals and number of articles included into the study 


JOURNALS 2011 2012 2013 2014 2015 2016 Total 
(N) 
HEF 3 2 4 3 1 4 19 
KUYEB 1 4 4 3 2 1 13 
Education and Science 1 4 5 3 7 20 
TOTAL 5 6 12 12 6 12 52 


All articles published between 2011 and 2016 (N=1349) were reviewed thoroughly by each author 
individually. Total 52 articles were identified as research articles and others which are either irrelevant to ODL 
research or identified as other (editorials, book reviews, interviews, concept papers, position papers, reflection papers, 
field notes, technical notes, etc.). For this study, only the research articles were used. Zawacki-Richter’s (2009) 
classification was adapted due to the fact that the research categories and areas schema were more rigorous and based 
upon the views of international experts in the field of ODL. Categories related to methodology for coding largely 
follow those identified in Creswell (2012). When a study did not fit into the categories identified by Creswell, the 
name of the method that the writer of the article used was added into an existing category based on similarity of 
method with others in that category. 


Findings and Discussion 


In this study 1349 articles were used (Table 2). A total of 52 articles were examined through content analysis 
and the criteria determined. 


Keywords Indicated 

In this study totally 162 keywords were included and ranked according to their frequency. Figure 1 shows 
the most frequent keywords used in the articles examined. It can be seen in the figure that distance education, e- 
learning and blended learning are the most frequently used keywords in these articles. It may be concluded that as 
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parallel to the change in the international ODL research area, Turkish research area is also under the effect of online 
technologies. This conclusion is also in line with Bozkurt et al’s (2015) study. 


@ dsance education 
Bolearning 
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Figure I. Most Frequent Keywords 


Terms like web-based instruction, e-learning styles, learning object, learning styles, academic success and 
interaction are other mostly used keywords that this study concludes. This may show that the focus in the articles has 
been changed into learners. That is, with the development of open and distance learning instructional pedagogies 
change their shift from instruction to learners and how do they learn as an individual. 


Research Areas 

As has been mentioned before, this study used the classification developed by Zawacki-Richter (2009) to 
define the research areas in ODL. Among the micro level topics, interaction and communication in learning 
communities, and the learner characteristics are the most frequently studied ones. In the meso-level, educational 
technology is the only topic examined so often. Related to macro level, theories and models is the one investigated 
most often. 

These results support the literature such as Zawacki-Richter (2009) and Bozkurt et al (2015). Namely, it may 
be concluded that in the last 15 years trends and issues have changed in the same line globally. In other words, the 
researches conducted and published in Turkey show similar results in terms of trends and issues in the field of open 
and distance education. 

This current study examined totally 39 articles which are focused on open and distance learning research 
because of this limited number of articles the categories that are not mentioned above such as costs and benefits, 
quality assurance, innovation and change and so on are not researched in the articles that were published in these 4 
journals. This lack of research may be starting point for future researchers. 


Meso: Professional development... 
Micro: Interaction and... 
Macro: Research methods in... 


Macro: Theories and models 


Micro: Instructional design 


Micro: Learner characteristics 


Meso: educational technology 


Figure 2. Research areas 
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Theoretical/Conceptual Backgrounds 

Figure 3 shows the most frequent theoretical/conceptual backgrounds that were used in the studies conducted 
in Turkey. It is clear in the figure that blended learning, hybrid learning, and web-based learning are very common 
among researchers in Turkey. It can be concluded that the integration of technology to traditional education has also 
gained importance as a new trend. 

What is more, theories such as community of inquiry, collaborative learning, social learning theory and 
activity theory that explain how learning occurs on networks through collaboration in community cannot find a place 
themselves in the studies. However, Bozkurt et al (2015) stated that these theories are favoured by ODL researchers 
in their study. That means there is a different trend in terms of theories between researches abroad and researches in 
Turkey. This may also serve as a starting point to the researchers in Turkey and may help them to find a research area 
to be examined. 
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Figure 3. Theoretical background 


Research Designs 

Figure 4A and 4B shows the research methods and designs used in the articles examined. Qualitative, 
quantitative and mixed research designs are also used in this study that is the same in many other similar fields. As 
shown below, quantitative (55,3%) studies are the most conducted ones and then qualitative (34,2%) studies take the 
second place and as last research design Turkish researchers employed mixed (10,5%) research design as a method. 

These results are not in the same line previous study (Bozkurt et al., 2015) which shows qualitative (47%) 
studies in the first rank and then quantitative (37%) studies except for mixed (16%) research design. On the other 
hand, Zawacki-Richter et al’s study (2009) reported similar results in terms of ranking. It is interesting to note that, in 
all these 3 studies mixed research design remains as the least frequent research design. That is, from 2000 to 2015 
there is no significant change in the research design trends of the articles. 


@ Mixed 
@ Qualitative 
@ Quantitative 


55.3% 


Figure 4. Research method 
Of all researches experimental, descriptive and case study designs are the most frequent research design 


models in the articles examined. This data reveals that there is a need to conduct more studies employing different 
research design models. 
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Figure 5. Research designs employed 


Data Collection Tools and Strategies 

This study also investigated data collection instruments and tools. It is clear in the figure that scale is the most 
frequent data collection tool. Questionnaire, achievement tests and interviews are also in common. Bozkurt et al (2015) 
also reported questionnaires and interviews as the most frequent data collection tools. In other words, the research 
trends somehow follow a similar path in the field of open and distance education. 
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Questionnaire Test Interview 
Scale Achievement tests Observation 


Figure 6. Data Collection Tools 


Variables 

As it can be seen in Figure 8, student achievement (29,2%) is the most frequent variable of the studies. On 
the other hand, these results overlapped with the results of keyword analysis which reports the change in the focus 
from teachers to learners. As students are in the center of research, their success and feelings are also become 
important. This may be concluded as a trend of Turkish research filed. 
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Figure 7. Variables 


Participant Groups 
The research question regarding to participants reveals a very clear result. According to this result, learners 
(75%) are the most frequent participants of the studies conducted in Turkey. Of all studies, very few studies benefited 
from teachers, managers, graduates or so on. This result is also in the same line with other studies conducted abroad. 
It is also interesting to note that, K-12 learners are not so common in the studies. Although there is an open 
education high school which has 945.390 students in Turkey, there is no study conducted or published in the Turkish 
journals. This may also be a point of interest in the future studies to be conducted in Turkey. 
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Figure 8. Participants 


Authors Cited Often 
This study also determined the leading contributors in the field according to their number of citations. Totally, 
there are 3106 citations.22 authors who have been cited at least 7 times are included in the table. 
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Table 3. Authors cited the most 


The First Author of the reference Frequency of citation 
Moore, M.G. 22 
Biiyiikéztiirk, S. 19 
Garrison, D.R. 19 
Brusilovsky, P. 17 
Fraser, B.J. 17 
Tsai, C.C. 17 
Horzum, M.B. 16 
Askar, P. 15 
Akkoyunlu, B. 13 
Gunawardena, C.N. 12 
Anderson, T. 11 
Baki, A. 11 
Jonassen, D.H. 11 
Chen, Y.J. 10 
Jung, I. 10 
Trinidad, S. 10 
Yildirim, A. 10 


References Cited Often 

In order to provide a comprehensive list for further research, this part of content analysis presents the most 
cited works (articles, books, web cites, etc.) within the articles analyzed in this study. It also presents a good reading 
list that may help researchers as a reference guide. This table was created by filtering 1635 references that have been 
cited at least 1 time. Studies that have the same rank were ordered according to their publication dates. It was 
interesting to note two points: First, the Turkish researchers cited the references related to the research methods more 
often than the references related to the field of ODL. Second, the references are quite diverse. Namely, the researchers 
used variety of resources in their studies. 


Table 4. References cited the most 
Reference Frequency of 
citation 
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How to design and evaluate research in education 
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Summary and Future Implications 


The results of this study try to reveal the research trends and issues in open and distance learning emerging 
from scholarly publishing four national journals. This study also intended to present the current research direction for 
future research. The results, or lessons drawn from this study can be summarized as follows: 

e Technology integration to traditional education is gaining importance. 

e The focus of the research studies has shifted from teachers to learners. 

e Educational technology, learner characteristics as well as theories and models are the most common 
research areas. 

e Achievement is the most focused variable among Turkish researchers. 

¢ Quantitative research method is also the most popular research method. 

e = The references and authors cited in the articles are quite diverse from all over the world. 


Overall as one can easily conclude that the quantity of the articles directly related to ODL published in Turkey 
in Turkish is limited compare the long history of ODL practices in the country. Especially finding no articles directly 
related to ODL in one of the major Turkish journals was very disappointing. In terms of quality, there are also several 
issues need to be addressed: First, the keyword analysis has shown that blended learning, hybrid learning, and web- 
based learning are among the most frequently used concepts. It may be inferred as that the ODL literature in Turkey 
is still in awareness phase where the researchers and the practitioners are trying to differentiate the ODL 
implementations. Another interesting finding is about the research areas: Educational technology as a meso level issue, 
instructional design and learner characteristics as micro level issues seem to be the most often studies topics. Although, 
ODL is considered as a subset of the field of educational technology in some countries, many believe in (Aydin, 2011) 
that it is a separate and mature field whose roots lies in the openness in education movement (Ozkul, 2014). In Turkey, 
both in research and in practice it does not really considered as a separate field although the Universitelerarasi Kurul 
(UAK), an agency established to facilitate the cooperation among the higher education institutions in Turkey, listed 
ODL as a separate Associate Professorship Field. In the same list ODL is also listed as an expertise area in two 
different Professorship Fields: Computer Education and Instructional Technology (namely educational technology) 
and Higher Education Studies. A field simultaneously cannot be considered as a separate field of study and as subset 
of other fields. This even shows that the nature of ODL has not been understood by the academic community in 
Turkey. 

Regarding the results of the study, following implications can be taken into consideration for future studies 
and practices: 

e More systematic review of literature type of studies needed to be able to understand the trends in the field 
in Turkey 

e Especially doctoral research studies should focus more on mixed methods to train future researchers in the 
field with experience in diverse research methods. 

e More studies should be done on the macro level issues, such as equal education opportunity, social 
inclusion, etc. 

e Studies should also concentrate on faculty/facilitators/instructors. 

e Comparison studies (face-to-face vs distance/online) should be avoided and more studies should rather take 
the effective online pedagogies into consideration. 

e The practitioners should employ different and more innovative ODL strategies and tools into their ODL 
offerings. 
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Abstract 


The study was to examine the effectiveness of BlendFlex instruction in relation to student academic outcomes in 
mathematics. BlendFlex instruction offers additional flexibility for students to choose between face-to-face, online 
asynchronous and online synchronous instructions according to their needs and change delivery methods at any time. 
Final grades for students who enrolled in a math course that was offered using three different delivery methods, 
BlendFlex, face-to-face and online, during the AY 2016-2017 and AY 2017-2018 were collected. Course success 
rates including grades A, B and C were calculated to indicate student academic outcomes. Chi-square analysis was 
performed to examine if statistically significant differences existed between students who chose BlendFlex, face-to- 
face and online instructions when compared course success rates. The results showed that the course success rate of 
online instruction was significantly lower than BlendFlex and face-to-face instructions. In addition, gender was a 
significant factor influencing course success rates of these instructions. The course success rate of online instruction 
for female students was significantly lower than BlendFlex and face-to-face instructions. The results implicated that 
BlendFlex instruction produced positive student academic outcomes. Instructors do need to consider gender 
differences when offering courses with different delivery methods. 


Keywords: BlendFlex Instruction, Blended Instruction, Academic Outcomes, Course Success Rates, Mathematics 
Introduction 


Blended instruction is getting popular and offers several advantages over face-to-face and online 
instructions. As with face-to-face instruction, blended instruction provides direct contact with instructors and other 
students. Unlike face-to-face instruction, blended instruction allows students to work in a self-paced environment 
that accommodates other scheduling needs (Currie, 2017). Blended instruction, also referred to as hybrid instruction, 
was defined in varying degrees of specificity (University of Washington, 2012). Gomes (2015) stated that in blended 
instruction, instructors provide content via a combination of face-to-face and online instructions, delivering between 
30 percent and 79 percent of the course content online, with the remaining contact delivered through face-to-face or 
other non-Internet mediums. 


260 


The online portions of blended instruction can be completed on the student’s schedule; however, the face-to- 
face portions are at a set time and location, and therefore have the potential to create scheduling conflicts. Students who 
need the instantaneous feedback provided in the face-to-face portions may struggle in the largely self-guided online 
portions that afford little to no direct interaction with the instructor. Additionally, the required face-to-face portions of 
blended instruction can potentially negate any cost savings in travel, room and board, and the accommodation of 
employment schedules that are typically associated with online instruction. 

To better accommodate the evolving needs of students, BlendFlex instruction was pioneered predominantly by 
a technical college in Georgia. The college received a $2.622 million grant from the United States Department of Labor 
Employment and Training Administration on September 25, 2013 and developed the BlendFlex instruction to provide 
services to those students who would not have local access to the instructional content but might not be comfortable 
with the self-paced format and demands of an online instruction. BlendFlex is a form of blended instruction that melds 
the face-to-face, online and telepresence methods into a single methodology. Rather than being restricted to only one 
delivery method of interaction for the duration of a course, students enrolled in BlendFlex courses can, at any time, 
participate in as many, or as few, of the delivery methods as they wish. To further clarify, a student registered for a 
BlendFlex course can choose to attend an in-class session during the pre-determined class meeting times on one day, 
attend an in-class session at a different campus the next day, then completely bypass the scheduled meeting times and 
participate online on another day. The student need only inform the instructor of their wish to change their delivery 
method and continue completing assignments according to the course syllabus. 

The college has presented its BlendFlex instruction at various higher education conferences such as the 2016 
Southern Association for Community College Research conference (Quinn & Lee, 2016) and has been featured in the 
University Business Magazine as an honoree in their Model of Excellence recognition program (Durso, 2017). At the 
time of the current study there is very little research focused on BlendFlex instruction. With the attention that 
BlendFlex instruction is receiving, and in consideration of the fact that other institutions could replicate the instruction, 
a complete analysis of the academic outcomes is critical. 


The Current Study 


The purpose of the current study was to investigate student academic outcomes in BlendFlex instruction at a 
large two-year technical college in Gegoria as compared to face-to-face and online instructions. The college offers over 
120 programs of study that include short-term certificates, diplomas, and associate degrees. Subject range from 
traditional college courses such as English, Math, History, and Psychology to trade and industry focused courses such 
as Automotive Technology, Air Conditioning Technology, and Welding Technology. The college has averaged 6,008 
students per term since fall of 2016. The student population has averaged almost 35% male and just over 65% female 
since fall of 2016. 

BlendFlex instruction was launched on September 25, 2013 to provide multiple pathways for students in the 
college to take courses for healthcare training. Since its inception, BlendFlex instruction has expanded from healthcare- 
related courses to credit courses in other programs, non-credit adult education programs, and preparation classes for the 
GED® high school-equivalency test. The current study focused on general education courses, specifically Math XXXX 
due to the high number of sections offered by the college and the diverse student population registered for the course 
since it is a requirement for several different majors offered by the college. 

During AY 16-17 and 17-18, 2,749 students who registered for MATH XXXX that was offered using three 
different delivery methods, BlendFlex, face-to-face and online instructions. Of those 178 students participated in 
BlendFlex instruction while the rest were in either face-to-face (NV = 1485) or online instructions (V = 1086). Among 
the students who chose BlendFlex instruction, 49 were male (27.5%) and 129 were female (72.5%). In face-to-face 
instruction, there were 543 male students (36.6%) and 941 female students (63.4%). One student did not indicate his or 
her own gender. In addition, 274 male students (25.2%) and 812 female students (74.8%) chose online instruction (see 
Table 1). The average age of BlendFlex group (M = 27.29) was slightly younger than face-to-face (M = 27.59) and 
online groups (M = 29.96) (see Table 2). 


Table 1. Gender of Students by Delivery Method during AY 16-17 and 17-18 


Gender 
Delivery Method Male Female Other 
BlendFlex 49 (27.5% 129 (72.5% 0 (0% 
FTF 543 (36.6% 941 (63.4% 1 (0.1% 
Online 274 (25.2% 812 (74.8% 0 (0% 
Total 866 1882 1 
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Table 2. Age of Students by Delivery Method during AY 16-17 and 17-18 


Age 
Delivery Method M SD Min. Max. 
BlendFlex 27.29 8.96 17 62 
FTF 27.59 9.47 16 67 
Online 29.96 9.31 15 70 


Final course grades were collected to examine student academic outcomes between BlendFlex, face-to-face 
and online instructions. The following grading scale was used in the college (see Table 3). Most courses require 
students to earn a minimum grade of C to receive graduation credit for the course. Therefore, course success rates 
including grades A, B and C were calculated as student academic outcomes to answer the following two research 
questions. 

1. Were there any differences in course success rates between BlendFlex, face-to-face and online 

instructions? 

2. Was gender a significant factor influencing course success rates between BlendFlex, face-to-face and 

online instructions? 


Table 3. Grade Scale 


Letter Grade Description 

100-89.5 

89.4-79.5 

79.4-69.5 

69.4-64.5 

64.4 and below 

withdrawn before the middle of the semester 

P withdrawn with a passing grade at the time of withdrawal 
F withdrawn with a failing grade at the time of withdrawal 
incomplete 


“|2|2]2|™/0]o|a]> 


Results 


During AY 16-17 and 17-18, 178 students participated in BlendFlex instruction; 114 of them received 
grades A, B and C. According to the descriptive statistics shown on Tables 4 and 5, the course success rate of 
BlendFlex instruction (64%) was higher than face-to-face (60.6%) and online instructions (51.3%). A chi-square test 
was performed to examine if there were any statistically significant differences in course success rates between 
BlendFlex, face-to-face and online instructions. The p-value of the chi-squared test was .000 < .05, X? (2, N= 2749) 
= 25.931, which indicated that course success rates between BlendFlex, face-to-face and online instructions were not 
equal. Based on the z scores using the bonferroni's method for comparison of column proportions, the course 
success rate of online instruction (51.3%) was significantly lower than BlendFlex (64%) and face-to-face 
instructions (60.6%). Students who chose BlendFlex instruction (64%) performed slightly better than face-to-face 
instruction (60.6%), but there were no significant differences found between these two delivery methods. 


Table 4. Final Grades for Students by Delivery Method during AY 16-17 and 17-18 
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Delivery Final Grades 

Method A B C D F WP WE W I 
BlendFlex 29 50 35 7 28 17 5 5 2 
(N=178) | 16.3%) | (28.1%) | (19.7%) | 3.9%) | (15.7%) | (9.6%) | (2.8%) | (2.8%) | (1.1%) 
Online 242 172 143 48 202 103 123 31 22 
N = 1086 22.3% 15.8% 13.2%) | (4.4%) | (18.6%) | (9.5% 11.3%) | (2.9%) | (2.0% 


Table 5. Course Success Rates for Students by Delivery Method during AY 16-17 and 17-18 


Course Success Rates 
Delivery Method No (W, WP, WF, D, F, 1) Yes (A, B, C) Total 
BlendFlex 64 (36%) 114 (64%) 178 
FTF 585 (39.4%) 900 (60.6%) 1485 
Online 529 (48.7%) 557 (51.3%) 1086 


According to the descriptive statistics shown on Table 6, the course success rate of BlendFlex instruction 
for female students (65.9%) was higher than face-to-face (64.2%) and online instructions (50.5%). For male students, 
the course success rate of BlendFlex instruction (59.2%) was also higher than face-to-face (54.5%) and online 
instructions (53.6%). Chi-square tests were performed to examine if gender was a significant factor influencing 
course success rates between BlendFlex, face-to-face and online instructions. First, the p-value of the chi-squared 
test for female students was .000 < .05, X? (2, N= 1882) = 36.849, which indicated that course success rates for 
female students between BlendFlex, face-to-face and online instructions were not equal. Based on the z scores using 
the bonferroni's method for comparison of column proportions, the course success rate of online instruction for 
female students (50.5%) was significantly lower than BlendFlex (65.9%) and face-to-face instructions (64.2%). 
Female students who chose BlendFlex instruction (65.9%) performed slightly better than face-to-face instruction 
(64.2%), but there were no significant differences found between these two delivery methods. Second, the p-value of 
the chi-squared test for male students was .774 > .05, X? (2, N = 866) = .513. It indicated that there were no 
significant differences in course success rates for male students between BlendFlex, face-to-face and online 
instructions. 


Table 6. Course Success Rates for Female and Male Students by Delivery Method during AY 16-17 and 17-18 
Course Success Rates 

Gender Delivery Method No (W, WP, WF, D, F, I) Yes (A, B, C) Total 
BlendFlex 44 (34.1%) 85 (65.9%) 129 

Female FTF 337 (35.8%) 604 (64.2%) 941 
Online 402 (49.5%) 410 (50.5 %) 812 
BlendFlex 20 (40.8%) 29 (59.2%) 49 

Male FTF 247 (45.5%) 296 (54.5%) 543 
Online 127 (46.4%) 147 (53.6%) 274 

Discussion 


Several findings were discovered from the current study. First, according to the descriptive statistics, the 
course success rate of BlendFlex was higher than the other two instructions, which confirmed the positive results 
from the previous studies related to the comparison between BlendFlex and non-BlendFlex instructions (Durso, 
2017; Quinn & Lee, 2016). These two studies found that BendFlex students have achieved slightly higher course 
success rates than their peers. BlendFlex instruction combines the benefits of both face-to-face and online 
instructions to enhance student learning with additional flexibility. 

Second, the chi-squared test results showed that the course success rate of online instruction was 
significantly lower than BlendFlex and face-to-face instructions. This was similar to the results of previous studies 
comparing face-to-face and online instructions (Amro, Mundy, & Kupezynski, 2015; Flanagan, 2012). Amro et al. 
(2015) examined course final grades in face-to-face versus online college algebra courses at a college in south Texas 
and found that the average grade of face-to-face students was higher than that of online students. Flanagan (2012) 
had a similar finding. She compared final course grades between face-to-face and online courses using one-way 
single factor ANOVA and found that without considering gender, face-to-face students scored significantly better 
than online students did. In addition, in the current study, students who chose BlendFlex instruction performed 
slightly better than students who chose face-to-face instruction, but there were no significant differences found 
between these two delivery methods. This was similar to the results of previous studies comparing face-to-face and 
blended instructions (Adams, 2013; Blissitt, 2016; Tseng & Walsh, 2016). For example, Adam (2013) implemented 
a quasi-experimental, mixed-method, posttest design to compare academic outcomes and course satisfaction 
between students who received face-to-face (face-to-face lectures and lab sessions) and blended instructions (web- 
enhanced CD-ROM lectures). She found that there were no significant differences in academic outcomes related to 
cognitive and psychomotor domains between these two delivery methods. Blissitt (2016) used a quantitative, quasi- 
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experimental, nonrandomized control group, pretest-posttest design to compare academic outcomes and course 
satisfaction between face-to-face and blended instructions in an introductory nursing pathophysiology course. She 
discovered that there were no significant differences found in posttest scores between these two delivery methods. 
Tseng and Walsh (2016) compared students’ motivation, level of learning outcomes and skills, and learning 
achievements (final grades) in an undergraduate English literacy course using two different delivery methods, face- 
to-face and blended instructions. Although students in blended instruction reported significant higher motivation and 
higher levels of learning outcomes, Tseng and Walsh (2016) found no significant differences in final grades of 
students between these two delivery methods. 

Third, in the current study, gender was a significant factor influencing course success rate between 
BlendFlex, face-to-face and online instructions. The chi-squared test results showed that the course success rate of 
online instruction for female students was significantly lower than BlendFlex and face-to-face instructions. Female 
students who chose BlendFlex instruction performed slightly better than the ones who chose face-to-face instruction, 
but there were no significant differences found between these two delivery methods. No significant differences were 
found in course success rates for male students between these three instructions. This was similar to the results of 
Flanagan’s (2012) and Paden’s (2006) studies. Flanagan (2012) compared the gender differences in academic 
outcomes in a junior-level business statistics course that was offered using two different delivery methods, face-to- 
face and online. She included 53 females and 36 males in face-to-face instruction, and 68 females and 77 males in 
online instruction. She discovered that the academic outcomes (final grades) for female students significantly 
decreased from face-to-face to online instructions. However, there were no significant differences in academic 
outcomes found for male students between these two instructions. In addition, Paden (2006) examined the impact of 
three delivery methods (blended, face-to-face and asynchronous online) on student retention rates and academic 
outcomes in an introductory undergraduate mathematics course at a large private university. The results showed that 
females retained at a significantly higher rate than males in the blended instruction. 


Conclusions 


To conclude, there were three main findings from the current study. First, the descriptive statistics showed 
that the course success rate of BlendFlex was higher than the face-to-face and online instructions. Second, the chi- 
squared test results showed that the course success rate of online instruction was significantly lower than BlendFlex 
and face-to-face instructions. Students who chose BlendFlex instruction performed slightly better than students who 
chose face-to-face instruction, but there were no significant differences found between these two delivery methods. 
Third, the chi-squared test results showed that the course success rate of online instruction for female students was 
significantly lower than BlendFlex and face-to-face instructions. Female students who chose BlendFlex instruction 
performed slightly better than female students who chose face-to-face instruction, but there were no significant 
differences found between these two delivery methods. In addition, no significant differences in course success rates 
were found for male students between these three instructions. 

The academic outcomes of BlendFlex instruction were positive when compared to face-to-face and online 
instructions. The course success rate of BlendFlex instruction was slightly higher than face-to-face instruction and 
significantly higher than online instruction. BlendFlex instruction combines the benefits of both face-to-face and 
online instructions, which helps produce positive academic outcomes and enhance student learning. In addition, 
instructors do need to consider gender differences when offering courses with different delivery methods. The data 
collected for the current study were quantitative, so directions for future research could employ qualitative research 
methods (e.g., observation, interview and focus group) to explore more about BlendFlex instruction related to 1) 
student learning process, 2) student perception and satisfaction, and 3) gender differences. 
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Descriptors: Learning Analytics, Classification Framework 
Abstract 


Many researchers have studied learning analytics during the last decade. However, obstacles still exist that 
impede enhancement of the field. These include a conceptual misunderstanding of learning analytics, an imbalance 
of research scope, and an inclusion of unrelated research. In this study, we suggest a classification framework for 
research on learning analytics. We analyzed 608 articles from the literature and found patterns regarding the 
definitions of learning analytics, contexts of the studies, methodologies utilized for the studies, and scopes of the 
studies. Based on the patterns, we developed a classification framework, consisting of four layers: Foundation 
Layer, Environment Layer, Development Layer, and Application Layer. We classified articles on learning analytics 
for professional learning and identified research gaps. The results show that the research on learning analytics for 
professional learning is not balanced in terms of scope. The conclusions and limitations are discussed. 


Introduction 


Analytics as the science of analysis is not a new concept. Even before anno domini, people analyzed 
patterns of clouds and winds to predict weather. However, modern analytics can be interpreted differently. With the 
development of computing systems and advancement of statistics, analytics is based mostly on proven statistical 
models and data collected by computing systems. Given these technological and technical changes, it is generally 
taken for granted that modern analytics refers to “the process of developing actionable insights through problem 
definition and the application of statistical models and analysis against existing and/or simulated future data” 
(Cooper, 2012, p.3). Modern analytics began in the 1980s. Though some companies started to use computing 
systems to manage customer data in the 1970s, database marketing and customer relationship management software 
in the 1980s were the beginning of modern analytics. 

Technologies, such as the Internet, to increase accessibility of various people and reduction in costs 
associated with using analytics contributed to the expansion of the usage of analytics to educational contexts and 
made many researchers and practitioners in the field interested in analytics for education. To reflect such interests, 
researchers have introduced academic analytics (Campbell, DeBlois, & Oblinger, 2007; Goldstein, 2005; van 
Barneveld, Arnold, & Campbell, 2012) and learning analytics (Ferguson, 2012; Long, Siemens, Conole, & GaSevic, 
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2011). While academic analytics supports decision-making in regard to an academic organization and its 
performance, learning analytics more focuses on an individual student’s learning and performance. 

Learning analytics (LA) is analytics for supporting decision-making regarding learning and learning 
environments. During the past decade, researchers from various disciplines have studied LA. While the diversity of 
disciplines has the advantage of expanding the research and introducing different viewpoints, it can be an obstacle to 
interdisciplinary collaboration if the interpretation of a particular concept varies from discipline to discipline. For 
example, researchers in one field regard LA as an automated tool that supports learning, whereas researchers in 
another field see LA as a process to analyze data on learning and learning environments. Another problem is that 
there are imbalances. For example, researchers have paid less attention to the LA for professional learning. At this 
point, it seems meaningful to try to find patterns and gaps in LA research to overcome those problems. 

This study has three purposes. The first is to find patterns of research on LA in the following respects: 1) 
how researchers have defined LA, 2) in what contexts researchers have conducted their research, 3) what 
methodologies researchers have used, and 4) what the research scopes are. The second is to develop a classification 
framework for LA research based on the patterns. The last purpose is to identify gaps in research on LA for 
professional learning. The following section introduces a literature review on LA and classification frameworks for 
LA research. 


Literature Review 


In this section, we provide a review of the literature on classification frameworks for LA or LA research, 
consisting of three viewpoints. We discuss studies on the classification frameworks for LA or LA research based on 
these viewpoints. Later in this section, we introduce the research questions of this study. 


Environment Viewpoint 

The environment viewpoint focuses on environmental factors and conditions related to LA. Some 
researchers (Greller & Drachsler, 2012; Pefia-Ayala, 2018) investigated LA or LA research from the environment 
viewpoint. Pefia-Ayala (2018) proposed a classification framework for LA research, for which the researcher 
explained three key areas: profile, applications, and underlying factors. In the framework, the researcher viewed 
legal issues, theoretical topics, and learning paradigms and settings as underlying factors influencing LA. The 
researcher also classified the definition of LA, stakeholders, field evolution, underlying domains, related domains, 
specialized lines, and prior reviews of the LA field as profile that reveals “an overall perspective of what LA is” 
(Pefia-Ayala, 2018, p. 4). Greller and Drachsler (2012) also considered the environment factors and conditions in 
designing a framework for the domain of LA. Their framework consists of six dimensions: stakeholders, internal 
limitations, external constraints, instruments, data, and objectives. The first five dimensions are associated with 
environments surrounding LA while the objective is closely related to the application of LA. The frameworks of 
Greller and Drachsler (2012) and Pefia-Ayala (2018) well reflect theoretical fundamentals of LA research as well as 
environmental factors and conditions related to LA. 


Development Viewpoint 

The development viewpoint focuses on elements that are necessary for developing LA. Aljohani et al. 
(2019), Muslim, Chatti, Bashir, Varela, and Schroeder (2018), Yassine, Kadry, and Sicilia (2016, April), and 
Ifenthaler and Widanapathirana (2014) studied frameworks for LA and we categorized them as the framework 
reflecting the development viewpoint. While the framework by Ifenthaler and Widanapathirana (2014) covers a 
broader scope and more elements of LA, the others emphasize the core functions of LA. 

Aljohani et al. (2019) proposed a course-adapted student learning analytics framework, which consists of 
four levels: instructor, data, data analytics, and presentation levels. This framework focuses on the analytical process 
from collecting data on learning and learning environments through presenting information such as feedback. 
Muslim et al. (2018) also focused on the analytical process in developing a modular framework for open learning 
analytics, but their framework includes more detailed processes and interactions among modules of the analytical 
process. The modular framework consists of four modules: analytics engine, analytics modules, analytics methods, 
and visualizer. Yassine, Kadry, and Sicilia (2016, April) considered user activities and learning outcomes in addition 
to data analysis and visualization in their framework. Their framework contains the definitions of data on user 
activities, mapping activities with learning outcomes, analysis of data on activities and learning outcomes, and 
information visualization. 

Unlike the aforementioned researchers, Ifenthaler and Widanapathirana (2014) introduced a holistic 
framework for LA, in which three core engines (learning analytics engine, personalization and adaptation engine, 
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and reporting engine), use various data generated by different sources (individual characteristics, social web, 
physical data, curriculum, and online learning environment), to provide information to the stakeholders including 
institution or governance. Though the holistic framework includes only institution and governance as stakeholders 
and the three engines in the framework are overlapped (e.g. visualization) and less relevant to the core features of 
LA (e.g. gamification), it has contributed to the field of LA in that it covers various data sources (e.g. social web and 
physical data) and separates functions of LA into learning analytics engine, personalization and adaptation engine, 
and reporting engine. 


Application Viewpoint 

The application viewpoint focuses on the uses and practical applications of LA. Many researchers in the 
field of LA studied the applications of LA and their studies highlighted specific purposes of LA, such as prediction 
of learning performance and retention (Hicks, 2018; Lu et al., 2018; Marbouti, Diefes-Dux, & Madhavan, 2016; Yu 
et al., 2018) or understanding of learners’ behaviors (Berland, Martin, Benton, Smith, & Davis, 2013; Martin-Monje, 
Castrillo, & Mafiana-Rodriguez, 2018; Ruipérez-Valiente, Mufioz-Merino, Leony, & Kloos, 2015), rather than 
discussing classification of applications of LA. There is not enough research on classification frameworks for LA 
seen from the application viewpoint. For this reason, we expanded our literature review to other data analytics areas. 

Some researchers (Fleckenstein & Fellows, 2018; Kumar, 2017; Skourletopoulos, Mastorakis, 
Mavromoustakis, Dobre, & Pallis, 2018) argued that analytics can be classified as descriptive, diagnostic, predictive, 
and prescriptive analytics based on the types of analytics applications. Their studies classified the types of analytics 
applications based on what kinds of information analytics can provide. Descriptive analytics shows information 
describing what happened or what is happening. Diagnostic analytics explains causal relationships by reporting 
information on why it happened or what it is happening. Predictive analytics provides information on what might 
happen. Prescriptive analytics recommends interventions based on the prediction. 


Rationale of the Research and Research Questions 

While there have been a few articles on classification frameworks for LA, frameworks for LA research 
remain under-researched. To promote the advancement of research on LA, it is necessary to develop a classification 
framework for LA research. Given the purposes of the study mentioned in the previous section, we addressed the 
following research questions: 1) what patterns exist in the definitions of LA, 2) what patterns exist in the contexts of 
research on LA, 3) what patterns exist in the methodologies used for research on LA, 4) what patterns exist in the 
scopes of research on LA, 5) how can the classification framework for LA research be developed, and 6) what are 
some gaps in research on LA for professional learning? In order to answer these research questions, we employed 
the research methodology illustrated in the following section. 


Research Methodology 


This study is a mapping review, which is a type of study that seeks “to map out and categorize existing 
literature on a particular topic, identifying gaps in research literature from which to commission further reviews 
and/or primary research” (Grant & Booth, 2009, p. 97). For the mapping review, we searched multiple academic 
databases to find articles to be reviewed. In this section, we explain our research methodology by describing articles 
selection process, information sources and search strategy, and exclusion criteria. 


Articles Selection Process 

For this mapping review, we searched for articles from five academic databases: ERIC from EBSCOhost, 
PsycINFO, Academic Search Complete from EBSCOhost, Education Research Complete from EBSCOhost, and 
Web of Science from Clarivate Analytics. We searched these databases on August Ist, 2018 and found 1,467 articles 
in total. After removing 754 duplicates, 713 articles remained. By applying the exclusion criteria explained later in 
this section, 608 articles remained for review. Figure / illustrates a summary of articles selection process for this 
study. 


Application of 
exclusion 
criteria 


Database Duplicate 


Search elimination 


Figure 1. Summary of articles selection process 
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Information Sources and Search Strategy 
As mentioned above, we used five academic databases as information sources for this study. Since the 

search feature of each database is slightly different, we used different search conditions for each database. Table 1 

shows the respective search conditions for each database. 


Table 1. Search conditions for each database 


No. of 
Database Search conditions articles 

ERIC from - Search keywords: ‘learning analytics’ for title or ’learning analytics’ for 157 
EBSCOhost KW identifiers 

- Only peer reviewed 

- Only journal articles for publication type 

- Find all my search terms 
PsycINFO - Search keywords: ‘learning analytics’ for title or ’learning analytics’ for 167 

keywords 

- Only peer reviewed 

- Only journal articles for document type 
Academic Search = - Search keywords: ‘learning analytics’ for title or "learning analytics’ for 249 
Complete from KW Identifiers 
EBSCOhost - Only peer reviewed 

- Only journal articles for document type 

- Find all my search terms 
Education - Search keywords: ‘learning analytics’ for title or ’learning analytics’ for 378 
Research KW Identifiers 
Complete from - Only peer reviewed 
EBSCOhost - Only journal articles for document type 

- Find all my search terms 
Web of Science - Search keywords: ‘learning analytics’ for title (TI) or learning analytics’ for 516 
from Clarivate topic (TS) 
Analytics - Only Articles for document type 


- Only from Web of Science Core Collection 
- Only from SCIE, SSCI, A& HCI, and ESCI (Emerging Sources Citation 
Index) 


Exclusion Criteria 

Given the purposes of the study, we adopted a comprehensive strategy in setting exclusion criteria. We did 
not include the quality of an article or a journal in the criteria. We excluded articles that are not written in English, 
not related to LA, or not research (e.g. editorial, commentary, or book review). Based on the exclusion criteria, we 
excluded 26 articles not written in English, 38 articles not related to LA, and 41 articles that are not research. 


Results 


Research Question 1: What patterns exist in the definitions of LA? 

Among 608 articles, we analyzed 185 articles that defined LA or used existing definitions of LA and found 
31 different definitions of LA. We included only six definitions in our analysis based on the frequency of the 
definition usage and excluded 25 definitions as they have been used only one time. Table 2 illustrates the definitions 
of LA that have been used in the articles we reviewed more than one time. 
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Table 2. Definitions of learning analytics and frequency 


Author(s) and 
publication year 


Definition 


No. of usage 


Society for Learning Measurement, collection, analysis and reporting of data about learners n= 154 

Analytics Research and their contexts, for purposes of understanding and optimizing (77%) 

(2011) learning and the environments in which it occurs 

Siemens (2010) Use of learner-produced intelligent data and analysis models to n=12 
discover information and social connections and to predict and advise (6%) 
on learning activities 

Slade & Prinsloo Collection, analysis, use, and appropriate dissemination of student- n=3 

(2013) generated, actionable data with the purpose of creating appropriate (1.5%) 
cognitive, administrative, and effective support for learners 

Lawson, Beer, Rossi, Collection and analysis of data in education settings in order to inform n=2 

Moore, & Fleming decision making and improve learning and teaching (1%) 

(2016) 

Willis, Zilvinskis, & Process of using live data collected to predict student success, promote n=2 

Borden (2017) intervention or support based on those predictions, and monitor the (1%) 
influence of that action 

Brown (2011) Collection and analysis of usage data associated with student learning n=2 
and its purpose is to observe and understand learning behaviors in (1%) 


order to enable appropriate interventions 


The most used definition of LA is “the measurement, collection, analysis and reporting of data about 


learners and their contexts, for the purpose of understanding and optimizing learning and the environments in which 
it occurs” (Society for Learning Analytics Research, 2011). 77% of the used LA definitions referenced the definition 
of the Society for Learning Analytics Research. Siemens’s definition of LA (2010) is the next most frequently used 
definition (6%), followed by Slade and Prinsloo’s definition (1.5%), Lawson, Beer, Rossi, Moore, and Fleming’s 
definition (1%), Willis, Zilvinskis, and Borden’s definition (1%), and Brown’s definition (1%). 

These definitions share commonalities. All the definitions consist of three components: data type, process, 
and purpose of LA (see Table 3). First, the definitions, except for the definition by Willis et al. (2017), clearly 
indicate that the data for LA are associated with learners, their learning, or their contexts. Second, the definitions 
describe the process related to LA. The process includes collection, analysis, and reporting steps. While the 
definitions by Siemens (2010) and Willis et al. (2017) do not specify steps during the process, the other definitions 
indicate each step during the process in a sequential manner. Lastly, all the definitions include purposes of LA. 
While the definitions by Siemens (2010), Slade and Prinsloo (2013), Willis et al. (2017), and Brown (2011) include 
purposes that are directly associated with functions of LA, the definitions by the Society for Learning Analytics 
Research (2011) and Lawson et al. (2016) include indirect purposes, such as ‘optimizing learning and the 
environments’ and ‘improving learning and teaching.’ 


Table 3. Data types, processes, and purposes in LA definitions 


Author(s) and 


publication year Data type Process Purpose 
Society for Data about Measurement, Understanding and optimizing learning and the 
Learning Analytics learners and collection, analysis environments in which it occurs 
Research (2011) their contexts and reporting 
Siemens (2010) Learner- Use of ... data and To discover information and social connections 
produced analysis models and to predict and advise on learning activities 
intelligent data 
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Slade & Prinsloo Student- Collection, analysis, Creating appropriate cognitive, administrative, 
(2013) generated, use, and appropriate and effective support for learners 
actionable data dissemination 


Lawson et al. Data in Collection and In order to inform decision making and improve 
(2016) education analysis learning and teaching 

settings 
Willis et al. (2017) Using live data _— Process of ... To predict student success, promote intervention 


or support based on those predictions, and 
monitor the influence of that action 


Brown (2011) Data associated Collection and To observe and understand learning behaviors in 
with student analysis of order to enable appropriate interventions 
learning 


Research Question 2: What patterns exist in the contexts of research on LA? 

We found 612 contexts from 608 articles and classified the contexts into five categories: higher education, 
K-12, MOOC-based learning, professional learning, and others. The ‘Others’ category includes articles that did not 
clearly indicate a context for research. Table 4 presents the results of our classification. As shown in Table 4, the 
most frequently studied context for LA research is higher education, occupying 300 out of 612 contexts, followed by 
K-12 context (n=69), MOOC-based learning context (n=25), and professional learning context (n=1 1). 


Table 4. Changes of contexts in LA research 


Context\Year 2011 2012 2013 2014 2015 2016 2017 2018* Total 


HE 1 6 9 20 52 62 91 59 300 
K-12 0 0 3 7 17 16 18 8 69 
MC 0 0 0 0 4 9 8 4 25 
PL 0 0 2 0 1 3 3 2 11 
Others 0 1 3 2 7 4 p) 3 25 
Total 1 14 27 51 110 147 170 92 612 


Note. HE = Higher education; PL = Professional learning; MC = MOOC-based learning. The figures for 2018 are 
the number of contexts found in the articles published between January and July 2018. 


Figure 2 illustrates changes in the relative ratio of the contexts in LA research reviewed. The results 
revealed that there is a significant imbalance in research on LA in terms of research contexts. Many researchers have 
studied LA in higher education contexts, but relatively few researchers have studied LA in other contexts, notably 
professional learning. The ratios for the higher education context have always been or higher than 64.3%. On the 
other hand, the ratios for the professional learning context have been less than 3.4%, except for 2013 (14.3%). 

In addition to the imbalance of contexts, the results discovered another pattern regarding the research 
context. Since 2015, researchers in the field have studied LA in the MOOC-based learning context. The relative 
ratios for the MOOC-based learning context in LA research have been 5.4% or higher. The last pattern found in the 
results is that the K-12 learning context has been steadily decreasing since 2014. 
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Figure 2. Changes in relative ratios of contexts in LA research reviewed 


Research Question 3: What patterns exist in the methodologies used for research on LA? 

We used a classification system of the SERVE Center (2008) to classify research methodologies that have 
been used for LA research. During the analysis of the research methodologies, we found articles that are hard to be 
classified by the classification system of the SERVE Center. Thus, we modified the classification system (see Table 
5). 


Table 5. Classification of research methodologies 


Research methodology Description 


Descriptive-qualitative Detailed descriptions of specific situation(s) using interviews, observations, 
and/or other qualitative data collection methods 


Descriptive-quantitative Numerical descriptions (frequency, average) 


Correlation/regression/and other Quantitative analyses of the strength of relationships between two or more 

association analyses variables using correlation analysis, regression analysis, likelihood ratio test, 
cluster analysis, random forest, path analysis, principle component analysis, 
analysis of variance, or other association analysis techniques 


Quasi-experimental Comparing an experimental group with a control group that is similar in 
characteristics but did not receive the intervention. Random assignment is not 
used to assign participants to an experimental group and a control group. 


Experimental Comparing an experimental group with a control group that is similar in 
characteristics but did not receive the intervention. Random assignment is 
used to assign participants to an experimental group and a control group. 


Meta-analysis Synthesis of results from multiple studies to determine the average impact of 
a similar intervention across the studies 
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Note. Adapted from the SERVE Center. (2008). Types of research methods. Retrieved from 
http://www.doe.virginia.gov/support/school_improvement/training/dta_student_support_sys/dropout_prevention/we 
binars_9-12/w2_s2_types_of_research_methods.pdf 


Table 6 shows the frequency of research methodologies found in LA research reviewed. The results 
revealed that substantial research on LA employed the descriptive qualitative methodologies (n=254) or association 
analysis (n=200). Less research on LA employed experimental (n=47) or quasi-experimental (n=35) methodologies. 
Figure 3 illustrates changes in the relative ratio of the methodologies in LA research reviewed. 


Table 6. Frequency of research methodologies in LA research reviewed 


Method 2011 2012 2013 2014 2015 2016 2017 2018* Total 

DQL 1 9 15 21 46 69 71 22, 254 
DQN 0 2 7 7 8 19 17 12 72 
CRA 0 3 B) 18 38 43 58 35 200 
QEX 0 0 0 0 5 7 14 9 35 
EXP 0 0 0 4 11 8 10 14 47 
Total 1 14 27 50 108 146 170 92 608 


Note. DQL = Descriptive-qualitative; DQN = Descriptive-quantitative; CRA = Correlational/regression analysis and 
other association analyses; QEX = Quasi-experimental; EXP = Experimental. The figures for 2018 are the number 
of research methodologies found in the articles published between January and July 2018. 
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Figure 3. Changes in relative ratios of research methodologies in LA research reviewed 
As shown in Figure 3, the majority of LA research (> 79% each year) between 2011 and 2013 employed 


descriptive methodologies, but the relative ratios of other research methodologies have steadily increased since 
2016. In particular, research using experimental or quasi-experimental methodology has been increased noticeably. 
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Research Question 4: What patterns exist in the scopes of research on LA? 

To classify the scopes of 608 research articles on LA, we developed an initial version of a classification 
framework based on our review of literature on the classification frameworks for LA or LA research. In developing 
the initial version of the classification framework, we included three layers: 1) Environment Layer, 2) Development 
Layer, and 3) Application Layer. During our review, however, we found scopes that are less relevant to these layers. 
Thus, we modified the classification framework and the final version of the classification framework consists of four 
layers: Foundation Layer, Environment Layer, Development Layer, and Application Layer. The Foundation Layer 
includes two scopes: 1) theoretical foundations of LA and 2) educational findings that can be used for LA. The 
Environment Layer includes three scopes: 1) legal and ethical environments, 2) technological environments, and 3) 
user's perception and behaviors. The Development Layer includes four scopes: 1) algorithm; 2) information 
presentation; 3) data collection, measure, and modeling; and 4) development methodology and process. Lastly, the 
Application Layer consists of four scopes: 1) descriptive LA, 2) diagnostic LA, 3) predictive LA, and 4) prescriptive 
LA. 

Table 7 presents the frequency of the scopes in LA research reviewed. The results indicate that the three 
most studied scopes of LA research are user’s perception and behaviors (n=102), theoretical foundations of LA 
(n=103), and educational findings that can be used for LA (n=108). On the other hand, the three least studied scopes 
of LA research are diagnostic LA (n=3), prescriptive LA (n=5), and predictive LA (n=12) in Application Layer. 


Table 7. Frequency of scopes in LA research reviewed 


Scope 2011 2012 2013 2014 2015 2016 2017 2018* Total 
APSC 0 0 0 2 0 0 2 1 5 
APRD 0 0 0 2 3 1 3 3 12 
ADGN 0 0 1 0 1 0 0 1 3 
ADSC 0 1 0 2 11 6 13 7 40 
DALG 0 0 2 2 18 15 18 21 76 
DNPR 0 2 5 1 10 11 15 8 52 
DDCM 0 2 3 3 9 9 13 5 44 
DDMP 0 2 1 2 4 8 10 4 31 
ELEE 0 0 1 2 1 15 2 2 23 
ETEN 1 1 3 7 14 15 17 13 71 
EUPB 0 2 2 8 16 25 29 20 102 
FTFL 0 6 8 15 20 19 26 9 103 
FEFL 0 1 3 7 14 38 33 12 108 
Total 1 17 29 53 121 162 181 106 670 


Note. APSC = Application Layer_Prescriptive LA; APRD = Application Layer_ Predictive LA; ADGN = 
Application Layer Diagnostic LA; ADSC = Application Layer_Descriptive LA; DALG = Development 

Layer Algorithm; DNPR = Development Layer_Information Presentation; DDCM = Development Layer Data 
Collection, Measure, Modeling; DDMP = Development Layer_Development Methodology and Process; ELEE = 
Environment Layer Legal and Ethical Environment; ETEN = Environment Layer_Technological Environment; 
EUPB = Environment Layer User's Perception and Behavior; FTFL = Foundation Layer_Theoretical Foundations 
of LA; FEFL = Foundation Layer_Educational Findings that can be used for LA. The figures for 2018 are the 
number of the scopes of research found in the articles published between January and July 2018. 


Figure 4 illustrates changes in relative ratios of scopes in LA research reviewed. As shown in Figure 4, 
research dealing with Foundation Layer or Environment Layer has always counted for more than half of the scopes. 
Relatively less research studied the scopes in Development Layer, but the interest in these scopes has continued 
since 2012. Lastly, little research investigated Application Layer. 
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Figure 4. Changes in relative ratios of scopes in LA research reviewed 


Research Question 5: How can the classification framework for LA research be developed? 

After developing an initial version of a classification framework for LA research, we improved it to 
accommodate new types of research scopes in LA research. In improving the framework, we focused on flexibility 
and universality to adapt to and accommodate the theoretical and technological changes in future. Figure 5 
illustrates the classification framework for LA research. To validate the classification framework, we classified 670 
scopes found from 608 articles into four layers and 13 types. 
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Application Layer 
Prescriptive LA 
Predictive LA 


Descriptive LA 


Development Layer 


Development Data Algorithm Information 
Methodology Collection, Presentation 
and Process Measure, 
and Modeling 


Technological User’s Perception Legal and Ethical 
Environment and Behavior Environment 


Foundation Layer 
Theoretical Foundations of LA Educational Findings 


Figure 5. Classification framework for LA research 


Research Question 6: What are some gaps in research on LA for professional learning? 

We mapped each of the research articles on LA for professional learning (n=11) with the classification 
framework for LA research. The results revealed that there are many research gaps in LA for professional learning 
(see Table 8). No research included in our review studied predictive LA, diagnostic LA, descriptive LA, algorithms 
for LA, development methodology and process, legal and ethical environments, and theoretical foundations of LA 
for professional learning. 


Table 8. Frequency of scopes in research on LA for professional learning 
Classification of LA research Frequency of scopes 

Application Layer Prescriptive LA 1 
Application Layer Predictive LA 
Application Layer_Diagnostic LA 
Application Layer_Descriptive LA 
Development Layer_Algorithm 
Development Layer _Information Presentation 
Development Layer_Data Collection, Measure, Modeling 
Development Layer Development Methodology and Process 
Environment Layer Legal and Ethical Environment 
Environment Layer_Technological Environment 
Environment Layer_User's Perception and Behavior 
Foundation Layer_Theoretical Foundations of LA 
Foundation Layer_Educational Findings that can be used for LA 


WOWrROONrFrCCC SO 
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Discussions 


Definition of LA 

We found 31 different definitions of LA from our review. Though there are some differences in the 
definitions of LA, the definitions share a few commonalities. However, there is a big difference between disciplines 
in defining LA. While researchers in the field of education take a broad approach to defining LA, ones in the field of 
engineering and business take a narrow approach. The former include analyzing data by human or a tool separated 
from a system that stores target data in LA research, but the latter consider LA research as one studying an 
automated tool that collects, analyzes, and provides data or information. Ifenthaler and Widanapathirana (2014) used 
the term ‘engine’ to mean the functions of the automated tool, but it is not sufficient to reflect the different views 
from the various disciplines. It seems there is a need to use the broad definition and narrow definition separately. 

Another issue regarding the definition of LA is that the most frequently used definition, the definition of 
Society for Learning Analytics Research (2011), has the following problems:1) It includes ‘measurement’ though 
measurement is not necessary for LA (Knight & Littleton, 2015) and 2) it includes ‘optimizing’ though the 
optimizing can be interpreted in different ways. At this point, thus, it seems there is a need to redefine LA to clarify 
its scope and purpose. 

We propose a broad definition and a narrow definition of LA. For example, a broad definition of LA refers 
to collection, analysis, and reporting of data or information on learners and their learning experiences to understand 
and improve learning. A narrow definition of LA refers to an automated system that collects, analyzes, and reports 
data or information on learners and their learning experiences to provide information supporting decision making 
regarding learning. 


Research context 

There is a significant imbalance in LA research in favor of the context of higher education. We assume that 
the imbalance probably occurred due to the relative ease of accessing data within the context. Thus, it seems 
necessary for researchers to collaborate with the stakeholders who are in the under-researched contexts. For 
example, researchers in the field of LA can study LA for professional learning more actively by collaborating with 
stakeholders in corporations. 


Research methodology 

Though there seemed to have been an imbalance of LA research in terms of research methodology in the 
early years of LA, the imbalance seems to be resolved by the researchers who employed experimental or quasi- 
experimental methodologies for their LA research. Comparing the contexts and scopes of LA research, the research 
methodology in LA research seems balanced. 


Research scope 

There is a significant imbalance in LA research in terms of the research scope. While considerable research 
dealt with the scopes in Foundation Layer and Environment Layer, little research covered the scopes in Application 
Layer. Thus, researchers in the field of LA can find relatively many opportunities for research. 

In addition, in the field of Multimodal Learning Analytics (MMLA), researchers have been interested in 
using a variety of types of data that can be used as input sources for LA. Some researchers (Abrahamson, Shayan, 
Bakker, & van der Schaaf, 2015; Lau et al., 2018; Lu, Zhang, Zhang, Xiao, & Yu, 2017; Munoz et al., 2018; Prieto, 
Sharma, Kidzinski, Rodriguez-Triana, & Dillenbourg, 2018; Zaletelj & KoSir, 2017) used data from sensors; such as 
electroencephalogram (EEG) sensors, motion sensors, and eye-tracking sensors; as data to be analyzed by LA 
engines. However, the multimodality can be applied for reporting information as well as collecting data. For 
example, future research on MMLA may focus on application of text-to-speech technologies and sonification to 
report information. 

Lastly, researchers might benefit from expanding the types of collected data for LA. Currently, many 
researchers in the field focus on LMS data on learners’ activities. However, this approach will limit the 
understanding of learners’ learning experiences as there can be many other factors, such as learning materials, 
content of instructor’s feedback, and contents of conversations between peer learners, influencing the learners and 
their learning experiences. Thus, expanding the types of collected data for LA seems a good approach. Such 
expanded data types include social learning analytics (de Laat & Prinsen, 2014; Shum & Ferguson, 2012), learners’ 
gestures (Viswanathan & VanLehn, 2018; Zaletelj & Kosir, 2017), eye movement tracking (Abrahamson, Shayan, 
Bakker, & van der Schaaf, 2015), and electroencephalography (Lau et al., 2018). 
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Classification framework for LA research 

We developed the classification framework with a focus on flexibility and universality to respond to 
possible changes in regard to theory and technology in the future. However, other researchers in the field may find 
some chances to improve the framework. For example, the scopes in the Foundation Layer of the framework could 
be divided into more detailed scopes or add newly discovered scopes. 

For this review, we included articles only from academic databases. It might be meaningful to use Google 
Scholar, a search engine for scholarly literature, to include more research articles to validate our classification 
framework for LA research. 


LA research for professional learning 

As shown in Table 8, there are many research gaps in LA for professional learning. There might be many 
reasons regarding such gaps, but we assume that the relative ease of accessing higher education data for research is 
one of the most influential factors on this phenomenon. To resolve this, as mentioned earlier, it is necessary for 
researchers to collaborate with the stakeholders who are responsible for professional learning in various 
organizations. 

In addition, to promote research on LA for professional learning, studying relationships and interactions 
between LA and other systems, such as talent management system or succession planning system. Based on the 
understanding of the relationships and interactions, researchers may find more research opportunities. 


Limitations 

Though we conducted iterative analyses for classifying articles that couldn’t be clearly classified into a 
category, there can be a different view for the classification. Therefore, a further discussion needs to be made with 
regard to the validation of the classification framework for LA research we developed. 
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Introduction 


Online learning has grown exponentially due to its flexibility and rapid development of the Internet 
technology in the past decades. According to a recent report (Seaman, Allen, & Seaman, 2018) over six million 
students are taking at least one distance course, representing 31.6% of all students, and over three million students 
are taking exclusively distance courses, representing 14.9% of all students. However, retention rate and graduation 
rate in online programs have become a concern (B. Smith, 2010; Herbert, 2006; Heyman, 2010; James, Swan & 
Daston, 2016, Xu & Jaggars, 2014) and need further investigation. Online courses require high levels of self- 
regulation, motivation, time management and engagement from students in order to be successful in online learning 
(Azevedo, Cromley & Seibert, 2004; Yen & Liu, 2009). If students are fully engaged and actively involved in 
course learning activities and stay connected with other stakeholders in online learning, it is likely they will achieve 
the desired learning objectives, complete their courses, and finish their online programs. Therefore, it is critical to 
investigate which types of student engagement in online courses contribute to student satisfaction and their 
perceived learning. 

The findings of this study revealed that some of the student engagement factors contribute positively to 
student satisfaction and perceived learning in online courses. The findings provided online faculty and instructional 
designers insights to consider incorporating different engagement factors when designing and revising their online 
courses. 


Student Engagement in Online Courses 


Student engagement has been defined in the literature as investment or commitment (Marks, 2000), 
participation (Kuh, Kinzie, Buckley, Bridges, & Hayek, 2007), or effortful involvement in learning (Pekrun & 
Linnenbrink-Garcia, 2012; Reschly & Christenson, 2012). Literature on student engagement has been focused on 
three aspects: behavioral engagement, emotional engagement, and cognitive engagement (Fredricks et al, 2004). 
Behavioral engagement refers to student participation in learning. In online courses, students actually complete 
certain online learning activities such as participating in discussions by responding to the original questions, peers’ 
posts, and/or asking challenging questions about the topic being discussed. Emotional engagement refers to positive 
and negative reactions to professors, peers, academics, and schools (Lee & Smith, 1995; Stipek, 2002). In online 
courses, students are emotionally associated themselves toward learning. They may simply like the topics, the 
activities, the instructor, and the students who interact with them or deeply appreciate the knowledge and skills 
gained in the learning process. Cognitive engagement refers to student effort, to what extent students are engaged in 
their course work to master complex ideas and difficult skills (Fredricks, Blumenfeld, & Paris, 2004). In online 
courses, students utilize varied strategies and efforts in online learning activities. Strategic students use 
metacognitive strategies to plan, monitor, and evaluate their cognition when completing tasks (Pintrich & De Groot, 
1990; Zimmerman, 1990). Students who are cognitively engaged use strategies associated with deep learning 
(Fredricks et al, 2004) such as reflective learning, live cases, group work, simulations etc. Reflective learning and 
live cases are among the most engaged learning activities that result in deep learning. 

Social engagement has also been proposed in the literature. It refers to the social aspect in a student’s 
collegiate experience (Knight, 2013). It is similar to the social presence as described in the three presence (Garrison, 
Anderson, & Archer, 2000) of online learning. In online courses, faculty can encourage students to participate in 
online forums such as “water cooler” or create “personal profiles” and share with the entire class to enhance social 
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engagement. Social engagement in online learning can also be further enhanced through “virtual clubs” or “virtual 
events”. All online students can choose to participate in campus events at a distance. 

Recent studies also identified additional aspect relevant to online learning that is collaborative 
engagement. This aspect is relevant to “the development of different relationships and networks that support 
learning, including collaboration with peers, instructors, industry, and the educational institution” (Redmond, et al, 
2018). In online courses, many learning activities such as collaborative projects, peer journaling etc. can be utilized 
to promote student collaboration. Tools like groups and Wikis in an LMS can be used to support these types of 
learning activities. 


Instrument of Online Student Engagement 


Students’ engagement in online learning has been measured in many different ways, ranging from self- 
report survey to observational measures. Since among these methods, self-report method is a cost-efficient and 
flexible way to gather a large quantity of data efficiently (Fowler, 2013), it was used widely in measuring students’ 
engagement in online context. Many instruments, such as surveys, questionnaires, or scales, have been established 
and used to assess the extent to which students engaged. For example, Dixson (2010) created the Online Student 
Engagement Scale to assess students’ behavioral, emotional, and cognitive engagement in online learning; and 
Coates (2006) established the Student Engagement Questionnaire to measure students’ engagement in aspect of 
behavior, emotion, and cognition. 

Although many instruments were created and used to measure students’ engagement in online learning, 
students’ engagement were measured in different aspects since these instruments were created according to different 
definitions of student engagement (Henrie, Halverson, & Graham, 2015). For example, Fredricks et al., (2004) set 
behavioral, cognitive, and emotional engagement as the indicators of the student engagement, while Handelsman, 
Briggs, Sullivan, and Towler (2005) constructed student engagement in skills engagement, emotional engagement, 
participation/interaction engagement, and performance engagement. Such examples highlight the unstructured 
construct of student engagement and the importance of providing well-defined construct. 


The Study 

This study attempted to validate the instrument to measure online engagement and determine whether 
relationships between types of online engagement and student satisfaction and student perceived learning exist and 
to what extent these online engagement factors contribute to student satisfaction and perceived learning. 

A variety of measures have been used to measure student engagement in literature. Student self-reporting in 
online surveys is used for this study as it provides data on how students are engaged in online courses and their 
satisfaction toward online courses and their perceived learning. A questionnaire was sent to all students taking four 
online graduate Educational Technology courses taught by one of the investigators in Fall 2018 and Spring 2019 
semester to collect data. Thirty-one responses were received from 36 students enrolled in those courses with a 
response rate of 86 percent. Many of the participants, 81% were female and 19% were male; 39% of them were 
master’s and 61% were doctoral students. All respondents had online learning experience as 87% of them had taken 
over 4 online courses before. 


Instrument Validation 

Upon reviewing student engagement instruments available, an instrument with five Likert- scale was 
created. The Liket-scale questions sought online student engagement level of each engagement aspect, and their 
satisfaction and perceived learning. Five questions regarding participants’ demographics were also included in the 
questionnaire. 

Prior to instrument evaluation, SPSS was used to screen the data. Each of the participant was assigned an 
ID number. Through checking the responses to 24 items of the OSEQ, the response of the ID #10 to the item 15 and 
the response of the ID #22 to the item 17 were missing. Therefore, these responses were coded as a missing data, 
which was coded as blank spaces in the data file. 

The OSEQ used in this study to measure online students’ engagement contained 24 items (Items | -24), 
each with four response options: | = “Strongly Disagree”, 2 = “Disagree”, 3 = “Agree”, and 4 = “Strongly Agree”. 
The direction of the rating scale suggest that higher scores represented higher levels of engagement in online 
learning. To ensure instrumentation quality, a standard Rasch analysis utilizing the WINSTEPS computer program 
was employed to evaluate both item functioning and dimensionality of engagement in online learning. The 
instrument performance was examined from the perspective of separation, categorical functioning, dimensionality, 
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item hierarchical order, fit statistics, which provided decisive information on the reliability and validity of the 
instrument used in this study. 

The instrument demonstrated a Rasch reliability of .92, which is high. High item reliability indicates that 
the sample size is large enough for stable comparisons between items. Hight item reliability also suggests that the 
instrument represented a clear line of inquiry, in which some items were more difficult and some items were easier, 
and that the confident could be placed in the consistency of the inferences of person ability, if these same items were 
given to another sample with comparable ability levels, is warranted (Bond & Fox, 2015). 

Separation is a measure of the spread of the estimates relative to their precision (Linacre, 2012a). The item 
separation value for the 24 items was 3.48, which was transformed into a strata index [Strata = (4G+1)/3; Wright & 
Masters, 1982] of 4.97 (rounded down to 4). This indicated that the four response categories were able to separate 
the participants endorsement of engagement into four statistically different groups. Good separation reflects small 
error, and the higher the separation is, the more confidence we can place in the replicability of item placement across 
other samples (Bond & Fox, 2015). 

To evaluate the reliability statistics, the non-extreme person estimates were used because it is more 
conservative (Wright & Stone, 1979). There were three participants classified as extreme respondents. WINSTEPS 
summary statistics of the 28 measured non-extreme participants shows that the real-person reliability was .86 and 
the model-person reliability is .88, which is not large difference from the real-person reliability. The Rasch 
reliability was considered to be ‘good’ (Miller et al., 2003) among participants. The real-person estimates were used 
to interpret the data because these estimates are more conservative. 

The Rasch person separation index is a reliability index determined on the basis of how many statistically 
different levels of engagement were distinguished by the items. The person separation of 2.49 transformed into a 
strata index [Strata = (4G +1)/3; Wright & Masters, 1982] of 3.65 (rounded down to 3). This indicates that three 
statistically different groups of participants on the OSEQ variable and it provides evidence that there is a 
quantifiable engagement measure. 

Rating scale functioning also contribute to instrument performance. If the rating scales function well, the 
step values should be arranged distinctively and monotonically. The summary of measured steps shows that step 
calibrations for Categories 1, 2, 3, and 4 progressed from the base line to -2.17, to -.52, and to 2.69. That means that 
the step difficulty between the response | (i.e., Strongly Disagree) and the response 2 (i.e., Disagree) is 2.17 logits, 
the step difficulty between the response 2 and the response 3 (i.e., Agree) is 1.65 logits (2.17-.52 = 1.65 logits), and 
the step difficulty between the response 3 and the response 4 (i.e., strongly agree) is 3.21 logits (2.69 — (-.52) = 3.21 
logits). Therefore, the step threshold increase met the recommended guideline of being above 1.4 logits meaning that 
respondents reliably distinguished between the rating scale categories. In addition, the average measure ranged 
between -3.39 logits to 3.81 logits, which increased with the category value. Overall, these findings suggest that the 
rating scale used on the instrument was functional for students. The step values were arranged monotonically along 
the linear measure and the scales were functioning reasonably well. However, the number of participants selecting 
category | was only 1% leaving almost a break, which may threaten the usefulness of this category. In addition, the 
probability curve shows that the peak of Category 2 was not distinctive enough. 

The difficulty difference in category “4” (Strongly Agree) between the easiest item and the most difficulty 
item was about three logits, and the difficulty difference between the steps of each item was not equal. The 
difference in step difficulty between “3” (Agree) and “4” (Strongly Agree) was more than five logits, while the 
difference in step difficulty between "2" (Disagree) and "3" (Agree) is about two logits. Thus, qualitatively, the raw 
score 4 of Item | (Explore online course site) was not equal to the raw score 4 of Item 18 (Build learning community 
to create sense of belongs), and category values were not intervals, because (4-3) # (3 — 2). 

Item Fit Statistics show that the measured infit mean square values (MNSQ) fell within the range of .60 and 
1.4 suggested as acceptable for Likert-type rating scales and the measured outfit MNSQ values fell within the range 
of .60 and 1.4 also suggested as acceptable for Likert-type rating scales (Bond & Fox, 2015). In addition, another 
statistic -Z-Standardized score (ZSTD) fell within the range of -2 and +2 also suggested as acceptable with a sample 
size of between 30 and 300 (Linacre, 2012a). Moreover, the point-measure correlation (PT- MEASURE) must be 
positive and larger than .3 were suggested as acceptable (Linacre, 2012a) because negative or ‘nearly zero’ values of 
PT-MEASURE correlations is the signal that items are problematic and are not consistent with the construct. 
Looking at the four indicators (Infit mean square, Outfit, mean square, Infit ZSTD, and Outfit ZSTD) and the point- 
measure correlation, item | (Infit MNSQ = 2.54 > 1.4, and Infit ZSTD = 3.5 > 2), item 3 (Infit MNSQ = 2.23 > 1.4, 
and Infit ZSTD = 2.9 > 2), item 4 (Infit MNSQ = 2.10 > 1.4, and Infit ZSTD = 2.5 <2), item 2 (Infit MNSQ = 
2.07 > 1.4 and Infit ZSTD = 2.4 > 2), item 23 (Infit ZSTD = -2.0), and item 22 (Infit ZSTD = -2.1 < -2.0) were 
detected as misfitting items and needed to be further investigated. 
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According to Bond and Fox (2007), when the MNSQ >1.4 or ZSTD > 2, it suggested that there is more 
variation than modeled; when the MNSQ < 0.7 or ZSTD < -2, there is less variation than modeled. So by looking at 
the item 1, 3, 4 and 2 closely, these items’ ZSTD is larger than 2, so these items are underfitting (1.e., the response 
pattern is too haphazard and the response to this item is too deterministic), which may degrade the quality of the 
ensuring measures. In addition, both item 22 and item 23’s ZSTD is less than or equal to -2, therefore, item 22 (I 
will take a similar online course again) and item 23 (“I have achieved the stated learning objectives for this course”) 
are overfitting (i.e., the response pattern is too determined and the responses to these items are too erratic), which 
might mislead us into concluding that the quality of our measures is better than it really is. Although item 1, 2, 3, 4, 
22 and 23 are misfitting according to the ZSTD, by investigating these items’ meaning, they are consistent with the 
theory. Therefore, these items were retained. 

To examine the dimensionality of the instrument used in this study, a Rasch Principle Components analysis 
of Residuals were conducted. The purpose of this analysis is not to construct variables, but to explain variance. The 
Rasch dimension explained 46.4% of the variance in the data. The explained variance is low (the cutoff value is 
60%). So, there is something else that influences the responses. Therefore, the analysis of the residuals was checked 
to find out if it is a random noise or systematic influencing. The largest secondary dimension, “the first contrast in 
the residuals” explained 13.6% (larger than 10%) of the variance, and the strength of the first contrast was 6.1 
eigenvalues, indicating that there was some unexplained variance. 

From the item-person map shows that the mean of measure of the student engagement rested about 2.5 
logits above that of the items, suggesting that the items were easy to agree with. The higher end of the distribution of 
students higher than the highest levels of items. This indicates that there are a group of students who are much more 
engaged than this instrument can measure with these items. Thus, more items that can measure the higher levels of 
engagement as needed, in order for students’ engagement to be measured more accurately. The difference in 
difficulty between the easiest item and the most difficulty item was about five logits (between -2.2 logits to 3 logits). 
The easiest item to say ‘Strongly Agree’ was located at the bottom of the item hierarchy, and it was item 2 (“TI 
participate in online activities (online discussions, online chat etc.) according to the course schedule.’’), 4 (“I log 
onto to the course and view course materials on a regular basis according to the schedule.”), and 9 (“I review my 
coursework for mistakes to improve it before submitting to the course site.”). The most difficult item to say 
‘Strongly Agree’ was located at the top of item hierarchy, and it was 18 (“I build or join learning communities to 
create a sense of belonging.”). Reviewing course material and completing course work were easy for participants to 
endorse, while creating sense of belongs was relatively hard for this group to endorse. In addition, there were 4 
students located on the top of the item hierarchy with engagements measures near 6 logits. This indicated that these 
students were likely to endorse all the items in this questionnaire. 


Data Analysis and Results 


Following the psychometric analyses of the data (i.e., the Rasch rating scale analysis), WINSTEPS person 
measures and category measures (logits) were used in place of standard raw scores for hypothesis testing to avoid 
methodological flaws associated with the use of raw scores. Person measures and category measures were intervals 
generated from the interactions of student endorsability, item difficulty, and step difficulty, and varied with item to 
item. 

Since the spearman’s correlation calculates a coefficient, which is a measure of the strength and direction 
of the association/relationship between two continuous or ordinal variables, in this study, this test was used to assess 
potential relationships between the five types of student engagement and student satisfaction and perceived learning 
in online courses, given the variables (i.e., behavioral engagement, emotional engagement, cognitive engagement, 
collaborative engagement, social engagement, student satisfaction, and perceived learning) are continuous. Figure | 
presents the statistics and level of significance illustrating nonparametric correlations between the variables. 

In this study thirty-one participants were recruited. Preliminary analysis showed the relationship to be 
monotonic, as assessed by visual inspection of a scatterplot. There was no statistically significant correlation 
between behavioral engagement and students’ satisfaction, rj.) = .301, p = .099 > .05. However, there was a 
statistically significant, moderate positive correlation between behavioral engagement and students’ perceived 
learning, rs) = .542, p = .002 < .05. 
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Nonparametric Correlations 


Correlations 
BehavioralE Emotional_En Cognitive_En Collaborative Perceived_Le 
ngag gag gag _Engag Social_Engag Satisfaction arning 
Spearman's rho Behavioral_Engag Correlation Coefficient 1.000 158 180 .140 303 301 542” 
Sig. (2-tailed) ‘ 396 333 452 .097 .099 .002 
N 31 31 31 31 31 31 31 
Emotional_Engag Correlation Coefficient .158 1.000 505” 476" 214 733" 756" 
Sig. (2-tailed) 396 : 004 .007 248 .000 .000 
N 31 31 31 31 31 31 31 
Cognitive_Engag Correlation Coefficient .180 505” 1.000 489" 365° 306 427° 
Sig. (2-tailed) 333 .004 ; .005 044 094 017 
N 31 31 31 31 31 31 31 
Collaborative_Engag Correlation Coefficient .140 476 489" 1.000 620" 486" 428° 
Sig. (2-tailed) 452 .007 .005 ‘ .000 .006 016 
N 31 31 31 31 31 31 31 
Social_Engag Correlation Coefficient 303 .214 365° 620° 1.000 338 258 
Sig. (2-tailed) .097 248 044 .000 . .063 161 
N 31 31 31 31 31 31 31 
Satisfaction Correlation Coefficient 301 733" 306 486" 338 1.000 773" 
Sig. (2-tailed) .099 .000 094 .006 .063 ‘ .000 
N 31 31 31 31 31 31 31 
Perceived _Learning Correlation Coefficient 542” 756" 427° 428° .258 773" 1.000 
Sig. (2-tailed) 002 .000 017 016 161 .000 i 
N 31 31 31 31 31 31 31 
**, Correlation is significant at the 0.01 level (2-tailed). 
*. Correlation is significant at the 0.05 level (2-tailed). 
Figure 1 


This test was also run to assess the relationship between emotional engagement and students’ satisfaction 
and perceived learning. Preliminary analysis also showed the relationship to be monotonic, as assessed by visual 
inspection of a scatterplot. There was a statistically significant, strong positive correlation between emotional 
engagement and students’ satisfaction, r(.) = .733, p <.001. Furthermore, there was a statistically significant, strong 
positive correlation between emotional engagement and students’ perceived learning, r(.) = .759, p <.001. 

In addition, this test was run to assess the relationship between cognitive engagement and students’ 
satisfaction, and perceived learning. Preliminary analysis showed the relationship to be monotonic, as assessed by 
visual inspection of a scatterplot. There was no statistically significant correlation between cognitive engagement 
and students’ satisfaction, ris) = .306, p = .094 > .05. However, there was a statistically significant, moderate positive 
correlation between cognitive engagement and students’ perceived learning, rj) = .427, p = .017 < .05. 

The Spearman’s rank-order correlation was run to assess the relationship between collaborative 
engagement and students’ satisfaction, and perceived learning. Preliminary analysis showed the relationship to be 
monotonic, as assessed by visual inspection of a scatterplot. There was a statistically significant, moderate positive 
correlation between collaborative engagement and students’ satisfaction, 1) = .486, p = .006 < .05. Furthermore, 
there was a statistically significant, moderate positive correlation between collaborate engagement and students’ 
perceived learning, rg) = .428, p = .016 < .05. 

The test was also run to assess the relationship between social engagement and students’ satisfaction, and 
perceived learning. Preliminary analysis showed the relationship to be monotonic, as assessed by visual inspection 
of a scatterplot. There was no statistically significant correlation between social engagement and students’ 
satisfaction, rs) = .338, p = .063 > .05. There was no statistically significant correlation between social engagement 
and students’ perceived learning either, rj) = .258, p = .161 > .05. 


Discussion, Limitation, and Future Studies 


As discussed in the instrument validation section, the instrument demonstrated a Rasch reliability of .92, 
which is quite high. The item fit statistics within the range of 0.6 and 1.4 indicated these items were acceptable for 
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Likert-type rating scales. However, some of the items with misfit score of 1.58 and 1.39 need to be revised. When 
further examining the items for cognitive engagement, some of those items actually resemble the theme of 
behavioral engagement. New items to better measure cognitive engagement need to be added. For example, “I 


integrate ideas from multiple sources when completing course assignments 


” 


“T justify my decisions with educational 


framework when completing course assignments” “I monitor my learning progress when completing my course 
work” and “T reflect on the course work I completed” need to be added to further investigate the cognitive dimension 


of student engagement. 


Figure 2 illustrate the relationships between each of the engagement factors and student satisfaction and 


perceived learning. 
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Figure 2 


Limitations of this study include small sample size and some items in the instrument do not quite measure a 
specific type of engagement. We will add more items to measure each type of engagement, and revise the items for 
the cognitive dimension and administer the survey to more online courses for analysis. 
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Introduction 


Computational thinking (CT) is defined as an analytical thinking approach and problem-solving process that 
breaking problems into small pieces, designing systems for possible solutions, and understanding the relationship 
among the pieces by drawing fundamental concepts (Wing, 2006, 2008; National Research Council, 2010a).When 
Wing's (2006, p. 33-35) pronouncement that “[computational thinking] represents a universally applicable attitude 
and skill set everyone, not just computer scientists, would be eager to learn and use" (p. 33), following this trend, the 
number of studies related to computational thinking and its’ implementations shows a huge increase. However, 
researchers are usually confused with the difference between CT and computer science (CS) concepts. Even though 
CT consists of some concepts that are the core of computing and computer science, it also includes efficient practices 
such as abstraction, decomposition, pattern generalization, and prediction. These concepts are also used by scientific 
and mathematical disciplines for modeling, reasoning, and problem-solving (National Research Council, 2008). 
According to Henderson et al. (2007), CT has been shown as the core of all science, technology, engineering, and 
mathematics (STEM) disciplines and it leads the idea of teaching computing in K-12 education. Also, most of the 
studies show teaching programming and CT skills is an effective method to cope with the challenging science and 
math concepts (Hambrusch et al., 2009, p. 183-187; Blikstein and Wilensky, 2009, p. 81-119). 

After Wing’s call to action for teaching CT skills to prepare students for being a productive member of the 
society, two National Academy of Sciences workshops were organized to find out the nature of computational thinking 
and its cognitive and educational interventions on K-12 curriculum (National Research Council, 2010b). These 
workshops also aimed to explain the pedagogical aspects of CT (National Research Council, 2011). In the first CT 
workshop, procedural thinking and programming (Papert, 1980, 1981, p. 82-92) were discussed as early concepts of 
CT. Even though this workshop seemed like a beginning process, it added significant value to examine and evaluate 
many concepts from computer science discipline to take CT beyond the idea of “just programming”. On the other 
hand, the workshop found out the lack of consensus that looks like to have bedeviled this field. Besides, it led to many 
unanswered questions: “How can CT be recognized? What is the best pedagogy for promoting CT among children? 
Can programming, computers, and CT be legitimately separated?” (National Research Council, 2010) 

While earlier researchers focus on the definitions of CT, since 2012, research interests related to CT have 
been shifted to work on these unanswered questions. Graver and Pea (2013, p. 38-43) explain that researchers have 
started to interest more practical questions such as how to encourage the implementation of CT in K-12 education and 
how to assess students’ CT abilities. In the last three decades, issues of how to teach and learn programming and CS 
constitute a large amount of the literature. Much of what we collectively know is largely based on Papert’s studies in 
the 1980s by using game-based learning environments using languages such as LOGO and BASIC. He indicates that 
“Twe] can give children unprecedented power to invent and carry on exciting projects by providing them with access 
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to computers, with a suitably clear and intelligent programming language and with peripheral devices capable of 
producing on-line-real-time action” (Papert, 1972, p. 245-255). Yet, providing all sources without knowing how to 
implement them in K12 settings to increase students’ motivation and learning achievement should be answered. 
However, the bulk of this CS education research is set in the context of undergraduate classrooms; little is known 
about how applications of game-based learning foster computational thinking in K-12, especially in elementary 
schools. 

Therefore, the purpose of this study is to obtain a better understanding for educators and researchers by 
systematically reviewing and synthesizing recent research since 2010 on CT regarding applications of game-based 
learning (GBL) in K-12. 


The following research questions are addressed in this study. 


RQI1: What range of topics does the current literature cover? 

RQ2: What kind of game-based learning environments and tools have been shown effective in promoting 
CT? 

RQ3: What are the reported outcomes related to students’ engagement, motivation, and achievement in CT? 

RQ4: How was the CT skills assessed in recent literature? 


Methodology 


Data Sources 

The main data sources include journal articles, conference papers, and doctoral dissertations. However, due 
to language barriers, only articles written in English were included. Also, the discipline criteria are limited to the field 
of education because of researchers’ academic background. The following keywords and their combination were used 
with regard to the main concepts of CT and GBL by using Boolean logic such as (Computational thinking AND (k12- 
K12 OR elementary school OR middle school)). 

The search was conducted in databases that are well-known and well-established in the field of computational 
thinking and game-based learning: ERIC (Education Resources Information Center), PsycInfo, IJGBL (International 
Journal of Game-Based Learning), IEEE (Institute of Electrical and Electronics Engineers), and Summon. Besides, 
Google Scholar was used for additional searches. 


Keywords: 

Computational thinking 

Computing education 

Game-based learning 

Games 

Gamification 

Digital games 

k12 or K12 or Elementary school or Secondary Education 
STEM education 

Non-STEM education 


Coding and Analysis 

The first step of the study identified 951 articles between 2010 and 2019 (see Table 1). At the beginning of 
the coding and analysis process, these studies were examined initially in regard to their titles, abstracts, and keywords. 
The initial review helped to see the relatedness of the studies at first. The criteria of relatedness were whether studies 
provided analytical results related to enhance computational thinking or not (see Table 2). 


Table 1. Distributions of studies in each database 


Database Number of Articles Identified 
ERIC 13 

PsyclInfo 3 

IJGBL 649 
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IEEE 99 
Summon 187 


Total 951 


In the selection stage, different frameworks were followed used by Zhang and Nouri (2019) and Sengupta et al. (2012). 


Table 2. Selection criteria 


Inclusion Criteria Exclusion Criteria 

K12 education (kindergarten through the 12th Other stages of education such as pre-university level, college 
grade (1-12)) students, and graduate students. 

Empirical studies Theoretical studies 

Studies that provide evidence about students’ Studies do not provide information about students’ learning 
learning outcomes outcomes 


Using educational tools or technologies to foster Studies not mention about any interventions 
computational thinking development 


Findings 


Computational Thinking Through Game Based Learning 

Game-based learning is defined as implementing game elements such as core mechanics, challenges, and 
goals into real-life settings to enhance learning. According to the study of the National Purchase Diary Panel (NPD) 
Group, in the United States, %82 of children ages between 2 to 17, approximately 64 million children, play video 
games (N.P.D Group, 2009). This study reveals how much games are attractive to children. Therefore, GBL as a 
teaching method seems promising to enhance students’ learning and motivation. Since an increase in implementation 
of CT pushes schools searching effective teaching methods (Gouws, Bradshaw, and Wentworth, 2013, p. 10-15), 
many researchers indicate that GBL, especially game design part, seems promising for teaching CT concepts in K-12 
settings (Baytak and Land, 2010; Rowe et al., 2017, p. 45). 

In the literature, different theoretical frameworks are promoted to guide students and teachers to improve CT 
skills through GBL. However, these frameworks show differences based on teaching or focusing particular skill(s): 
decomposition, pattern recognition, abstraction, algorithm thinking, and evaluation (for checking the CT skills see 
Selby and Woollard, 2013) in the development of CT. In Figure 1, Jiang et al., (2019, p. 29) suggest a theoretical 
framework for targeting coding environments to improve CT skills with regard to gameplay elements and design 
principles. In the framework, they put emphasis on computational problem-solving (CPS) that a basis aspect of CT, 
design of a puzzle game, and their relationship. 


Game Elements and Design Principles 


CPS Components What Players Have to Do 


formulate tachi 1.aware of being a problem-solver core 1.meta-gaming 
problems mechanics — 


_ 2.identify and abstract a problem nner fading 


build algorithmic _ 1 build solutions via programming 
i 2.program without coding 


challenges Encapsulation 


solutions 


1.solve the problem 


Figure I. Computational Puzzle Design (CPD) Framework (Source: Jiang et al.,2019, p. 29) 


290 


Many studies especially focus on block-based coding game designs, follow similar structured frameworks 
like CPD, even though they do not specifically mention their theoretical framework. In these studies, players’ tasks 
and game designs show differences but the relationship between CPS and game elements is similar (e.g. Troiano et 
al., 2019, p. 208-219;Kazimoglu et al., 2012, p. 522-531; Harrison et al., 2018, p. 134-138). 

Following a structured theoretical framework is not solely enough for the development of CT skills, game 
elements and design in regard to the target audience play a significant role in students’ learning outcomes. In the 
literature, many researchers develop their own game projects while others prefer to use already developed software 
such as LEGO Mindstorm, Scratch, and Light Bot. However, surprisingly, even though researchers develop their own 
game designs, still, they tend to use block-based game design methodology (e.g. Howland and Good, 2015, p. 224- 
240; Leonard et al., 2016 ;Bauer, Butler, and Popovic, 2017, p. 26; Jiang et al., 2019, p. 29). Bauer et al., (2017, p. 
26) explain their reason to develop their own game design as keeping a balance between exploration and guidance in 
the games. According to them, existing game designs (called Dragon Architect) that aim to improve CT include either 
open-ended exploration with little direct instruction or full of exploration with a lot of direct instruction. In their game 
design, players use code blocks to move the dragon character for solving the puzzle (see Figure 2). Similarly, Jiang et 
al., (2019, p. 29) promote the “LittleWorld” game that includes non-coding blocks and puzzle designs to foster 
students’ computational problem-solving skills (see Figure 2). They indicate that puzzle game designs are appropriate 
for enhancing logical thinking, problem-solving skills, and pattern recognition which CT aims to promote. 


Let's build something cool! Click Learn more code | to 


get new tools and abilities, Like moving up and down or 
rotating the camera. 


Figure 2. Examples of block-based game designs (Dragon Architect and LittleWorld) (Source: Bauer, 
Butler, and Popovic, 2017) 


Moreover, a few studies implement hands-on game-based methods and role-playing game designs to develop 
CT skills and to enhance student communication, reading, and writing skills. For example, Jagust et al. (2018, p. 1-5) 
developed unplugged game-based activities to teach CT skills. In their study, four case studies were conducted with 
different age groups. However, only three of them are related to K-12 students, so the details of the fourth case study 
wouldn’t be provided here. In the first case study, they redesigned the famous children games such as Battleship, the 
dot game, and Packman by using pencil and paper. In this CT activity, students completed basic paper programming 
activities like doing in Scratch. Similarly, in the second case study, students completed a life-size graph paper game 
by working either individualist or collaboratively. In the single player mode, a student should execute the given 
program (drawing a shape) in the playing board. In the two-player mode, one student takes a “programmer” role while 
other plays as “robot”. The programmer writes the codes to complete the given programming task while the robot 
executes these written commands. Furthermore, the third case study is called “The Network”. The purpose of this 
study is to integrate the networks problems and unplugged activities into the curriculum. 6th grade students completed 
these activities like in the first case study. 
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a 


Figure 3. The photos from the case studies (Source: Jagust et al., 2018) 


In addition to Jagust et al., (2018, p. 1-5), another GBL activity is conducted to explore the relationship 
between computational thinking and English literature in the K-12 curriculum by Nesiba, Pontelli, and Staley in 2015. 
In their study, existed English curriculum units are integrated with GBL methods and CT skills. For example, students 
practice their CT skills by telling the story of “Macbeth” novel, the story of a Scottish soldier who wants to be King 
without thinking any consequences during his journey (see Figure 4). At first, students select some scenes from the 
novel and sketch out them for preparing a storyboard. After they create their storyboard, they implement these scenes 
by using ToonDoo tool, online comic-creation tool. Even though this study does not include any problem-solving 
skills, students have to use their algorithmic thinking abilities to develop a detailed step by step scene based on their 
storyboard. In the literature, algorithmic thinking is described as a way of getting to the solution by defining clear 
steps (Selby and Woollard, 2013; Edwards, 2011, p. 58-67). Although students do not solve any problem in the study, 
they define each step clearly to complete their tasks. Thus, this study provides an exceptional example of how to foster 
algorithmic thinking without a problem-solving task. Besides, according to Nesiba, Pontelli, and Staley (2015, p. 1- 
8), students also use their abstraction skills while scanning their scenes and criticizing the significant passages. 
However, the abstraction component of CT skills is defined as identifying general principles to generate specific 
patterns (Shute, Sun, and Asbell-Clarke, 2017, p. 142-158). Therefore, their findings related to abstraction and 
decomposition is arguable since students do not focus on generating patterns in their study. 


Figure 4. A student's work from Macbeth story (Source: Nesiba, Pontelli, and Staley,2015) 


Overall, most of the studies in terms of implementation of CT through GBL in K-12 setting address coding- 
oriented education or STEM education by using block-based applications such as Scratch, LightBot, and Combats. 
Only a few studies aim to enhance students’ CT skills in social science classes. 


Students’ Engagement, Motivation, and Achievement in CT 

According to Kotini and Tzelepi (2015, p. 219-252), the way of enhancing students’ CT skills is complex 
and is dependent on students’ engagement and motivation. Also, they indicate that gamified learning activities aim to 
foster CT skills have a positive impact on students’ achievement. Following this trend, many studies show integration 
of educational games into the K-12 curriculum help to increase students’ attention, learning motivation, and positive 
attitude (Yang and Wu, 2012, p. 339-352; Bai et al., 2012, p. 993-1003;Triantafyllakos, Palaigeorgiou, and Tsoukalas, 
2011, p. 227-242). However, only a few studies mention about improvements in students’ motivation in the 
implementation of a game-based learning activity to foster students’ CT skills. One example from these studies is 
conducted by Fogli et al. in 2017. They studied with 18 secondary school students to measure the effectiveness of a 
game-based system (TAPASPlay), which is designed as a block-based environment to enhance students’ CT skills. 
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According to their study, there is evidence that TAPASPlay provides an enjoyable environment to improve students’ 
motivation and CT skills with regard to collaborative learning. Moreover, a study also shows that primary school 
students are highly motivated to learn algorithmic thinking with tangible objects by using block-based environments 
like BYOB or Scratch (Futschek and Moschitz, 2011, p. 155-164). These two studies advocate that learning CT skills 
through GBL, or tangible objects can increase students’ motivation and engagement. On the other hand, this kind of 
studies is insufficient to explain the source of students’ motivation by showing evidence. In other words, studies in 
the literature do not provide detailed information about either student get motivated by involving a game-based activity 
or they are motivated because of improving their CT skills. 

Furthermore, in terms of students’ achievements in CT, most of the studies only put emphasis on 
decomposition, algorithm design, and abstract thinking (Harrison et al., 2018, p. 134-138; Kazimoglu et al.,2012; 
Bauer et al., 2017, p. 26). Surprisingly, current studies do not usually mention about pattern recognition and 
generalization while talking about students’ CT skills. However, these two CT components are the basis for the 
development of other components like algorithm design (Ambrosio et al., 2014, p. 25-34). In general, more studies 
should be conducted to explain the effects of CT through GBL on students’ engagement, motivation, and achievement 
in particular CT components. 


Assessment and Evaluation Methods in CT 

Adaptation of CT into the school curriculum is essential to prepare students for 21st-century skills and to be 
productive members of the society. However, it requires an assessment framework to understand in what extent K12 
school students are successful in CT abilities and which kind of GBL methods can engage students’ CT skills. Also, 
Nitko and Brookhart, (1996) indicate that assessment aims to improve the learning process with regard to needs of the 
students, schools, and curriculum. Besides, assessment tools play a critical role to bring CT through GBL to K-12 
settings for supporting the use of programming in middle school (Werner et al., 2012, p. 215-220) (Grover and Pea, 
2013, p. 38-43; Grover, 2015, p. 15-20). However, a widely accepted assessment method is missing to measure the 
effectiveness of CT interventions in the literature (Shute, Sun, and Asbell-Clarke, 2017, p. 142-158). According to 
Settle et al., (2012, p. 22-27), the needs for an appropriate assessment method is a critical step to control the reliability 
and validity of interventions in the CT area. 

In the literature, most of the studies use interviews, pre and post-tests, project portfolio analysis, document- 
based analysis, design scenarios or the combination of these methods as assessment methods in the interventions of 
CT (Brennan and Resnick, 2012, p. 25;Bubica and Boljat, 2018; Mioto et al.,2019). These are the traditional ways to 
gain an understanding of students’ learning outcomes related to CT concepts. Additionally, Shute et al., (2017, p. 142- 
158) present two assessment methods: Scratch-based and game/simulation-based. According to their study, Scratch- 
based assessment also includes portfolio evaluation, interviews, and design projects in addition to multiple choice and 
open-ended quizzes. On the other hand, they bring up pattern analysis and web-like graphic reports in the 
game/simulation assessment method. This evaluation method is based on visual analysis of students’ progress. 

Furthermore, Roman-Gonzalez, Moreno-Leon and Robles (2019) propose a comprehensive evaluation model 
to assess students’ CT abilities in regards to Bloom’s taxonomy (see Figure 5). They suggest several tools such as 
diagnostic, summative, and formative-iterative. Each tool specifically addresses one level of CT. For example, 
diagnostic tools aim to measure the understanding level of CT while summative tools focus on learning objectives. 
According to Mioto et al., (2019), since formative and summative assessments put emphasis on students’ cognitive 
abilities, they are more efficient during the development of CT. However, a few studies use these evaluation tools and 
do not include concrete examples of how to combine these methods in educational settings (Roman-Gonzalez, 
Moreno-Leon and Robles, 2019) 

To sum up, both qualitative and quantitative methods are preferable for the researchers in the field. Only, a 
few studies get benefits from data visualization reports to measure students’ growth. The examples of how to 
implement different assessment methods into CT education can take place in the future studies. 
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Figure 5. Bloom's taxonomy and CT assessment tools (Source: Gonzales, Leon, and Robles, 2019) 


Limitations of the Study 


This study has two main limitations. Firstly, the review is only conducted searches in five databases and 
Google Scholar. Also, studies are only written in English and the field of education was eligible. Therefore, it might 
not represent all works in the research area. 

In addition, this review focuses on the studies which provide positive evidence to show how students’ CT 
abilities can be enhanced through GBL. Thus, this scope may not present a complete picture of the improvement of 
CT skills in K12 education. It might lead to publication bias as a limitation of this study. 


Conclusion and Discussion 


Space limitations of this proposal limit our ability to discuss our synthesis in detail, which we intend to 
present at the AECT conference organized by the four research questions followed by recommendations for future 
research. Below are some of the highlights for each research question. 

According to our findings, most studies have focused on either STEM education to children in regard to 
computational thinking. In other words, most of the studies consider STEM components to increase learning outcomes, 
students’ confidence, and improve their capacities in terms of enhancing computational thinking. However, studies 
mainly put emphasis on the science component even though they come up with a generalization in STEM. For 
example, Jenson and Droumeva, (2016, p. 111-121) start with STEM as the main concern but then, they make narrow 
the study by putting a spotlight on science concept. On the other, a few studies only consider all of the STEM subjects 
in their research methodology. For instance, Tsarava et al., (2017, p. 687-695) focus on science, technology, 
engineering, and math subjects by building a bridge between real-life problems and CT-based solving. They design 
eight lessons that consider different problems related to each STEM subject for 3rd and 4th graders in primary schools. 
Their study is one of the few studies that show how to foster computational thinking through game-based learning in 
different subjects. Nevertheless, more study, especially related to technology and engineering in STEM, should be 
conducted to guide teachers or policy makers on how to enhance CT through GBL in these subjects. Besides, the 
number of studies in non-STEM disciplines like literacy or social sciences is also a few. Researchers can consider 
working on how to develop a CT curriculum to enhance students’ communication, reading, and writing skills. 

In addition, from a game-based learning perspective, our findings show that most of the studies regard block- 
based environments to support CT. Most common trends in the studies in terms of block-based environments are 
Scratch Jr, MIT AppInventor, Game Maker, Code Combats, ColoBot, and LEGO Programs. These environments are 
usually used for teaching algorithm or creating lesson plans to blend in an ongoing lesson for enhancing CT. For 
example, Baraté, Ludovico, and Malchiodi (2017) used the LEGO-based music notation program to foster primary 
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school students’ CT abilities. The findings of these studies emphasize a strong relationship between basic object- 
oriented programming and computational thinking. Moreover, from a theoretical perspective, we found that several 
studies are based on constructionism, constructivism, project-based theory, and active learning theory. Future studies 
might want to consider the role of social cognitive theory, information processing theory, and cognitive learning 
processes on students’ CT abilities. Since CT includes relevant abilities such as decomposition, the process of 
debugging, testing and analysis, and control structures are related to these learning theories. 

Most recent addressing the CT assessment has used either student-created, or pre-designed programming 
artifacts to evaluate students’ understanding and use of abstraction, conditional logic, algorithmic thinking, and other 
CT concepts to solve problems. In addition, ideas of deconstruction, reverse engineering, and debugging was mostly 
used to assess children's understanding in computational contexts. 

The findings of this study also pointed out to the use of gamification as a way to foster students’ 
computational thinking abilities, engagement and motivation in addition to their academic achievement at early ages. 

The analysis of existing literature in this study demonstrates that using game-based learning as an 
interdisciplinary activity has the potential to foster computational thinking and develop 21st-century competencies 
such as critical thinking, problem-solving, creation, and innovation. However, current studies did not focus on the 
roles of complexity, metacognition, and fluid intelligence for facilitating CT abilities through GBL. Thus, we 
identified those areas for further studies. 
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Abstract 

Entrepreneurship education has received enormous interests since entrepreneurs are key drivers of 
business growth across the world. Entrepreneurial creativity is identified as core competency in large 
multinational corporations in Thailand because of unique combination of sense of ownership and innovative 
thinking. Thai government has launched Thailand 4.0 policy since 2016 to enhance creativity and technology 
competitiveness to towards innovation-driven or creative economy. However, the 2019 Global Innovation Index 
reported that Thailand ranked No. 43 from 129 countries in the world since Thai entrepreneurs and corporate 
entrepreneurs still have lower level of entrepreneurial creativity skill, judging from the fact that 90% of the 
products are not differentiated enough. An in-dept focus group of 8 respondents are conducted among young 
managers in a multinational company to understand the triggers and barriers of entrepreneurial creativity 
development and found that their entrepreneurial creativity skill is at moderate level since they are not confident 
with their skill, driven by current development activities, focusing on theoretical, one-way, passive learning 
experiences. They do not feel engaged, motivated and confident in transfer the knowledge to practice. Design 
thinking is well recognized as the creative problem-solving process though human-centered approach, 
simulating the journey of entrepreneurship and have been used in developing business education. Social media 
is one of the most used platforms for daily live and learning with a feature, delivering open and collaborative 
platform. Therefore, a learning design with design thinking process is recommended with social media 
technology to deliver engaging, experiential, and motivating learning experiences among young managers. 


Introduction 


Entrepreneurship and entrepreneurship education have received enormous interests among every 
national governments, academic institutions and business industry since entrepreneurs are key drivers of 
business growth, alleviating the slowdown of global economy. This is because entrepreneurs came into the 
business with unique mindsets and competencies from recognizing emerging business opportunities, initiating 
innovative ideas, creating business planning, gathering required resources and drive with full passion to achieve 
the goals with strong commitment and calculated risks. The results have proven in the developed nations like the 
United States, United Kingdom, Singapore and Japan where entrepreneurs in small and medium sized 
corporation contributed at least more than 50% of gross domestic products and generating more than 65-70% of 
the whole country’s employment(Yildirim, Trout, & Hartzell, 2019). 

In the corporate world, entrepreneurial competency is identified as core competency in large 
multinational corporations like Coca Cola, P&G, Unilever, S.C. Johnson and Sons since this competency refers 
to the ability of executives, managers or employees to have strong ownership and continue innovating in their 
areas of responsibilities by weighing between opportunities and calculated risks to achieve impressive business 
results. This can be called “Corporate Entrepreneurship” to change the employees’ mindset from traditional 
“wait and see” to more “proactive and aggressive” way of working(Barringer & Ireland, 2012). The expected 
skill within entrepreneurship is creativity and business ownership. While ownership is more the aspect of 
engagement and passion with roles and companies, creativity is one of the skills and attitude within the areas of 
entrepreneurship that can be cultivated within the organization. Creativity is part of the 4Cs’ learning skill 
category in the 21“ Century skill. 

Creativity is regarded as one of the most important and key success competencies among entrepreneurs 
to become successful in their ventures. Burns and Burns (2014) mentioned about the 5 qualities of entrepreneurs 
which are (1) Creative thinking and innovation (2) Drive for achievement (3) Independency (4) Self- 
directed/control (5) Awareness of calculated risks. Creativity encompasses the process from recognizing the 
opportunities to creating an innovative business models to start new ventures. This is consistent with what 
Barringer and Ireland (2012); Bolton and Thompson (2013)and Kuratko and Hodgetts (2004) have included 
creativity as key essential component of entrepreneurs which includes creative problem solving, creative 
thinking as well as creative design. 


298 


Thai government launched Thailand 4.0 policy in 2016 to increase creativity and technology 
competitive advantage to deliver innovation with the intention to move the country from middle-income and 
efficiency-driven economy to become innovation-driven or creative economy(Murnpho & Unaromlert, 2018). 
However, the 2019 Global Innovation Index (developed by Cornell University, INSEAD and WIPO) indicates 
that Thailand ranked No. 43 from 129 countries in the world. The best-in-class in Asia is Singapore (8"), Korea 
(11), Hong Kong (13"), China (14") and Japan (15"). Another study done by Organization for Economic 
Cooperation and Development (OECD) in 2018 found that Thailand ranked No. 3, following Singapore and 
Malaysia from 10 countries in Southeast Asia on the aspect of innovation and technology outputs(The 
Secretary-General of the OECD, 2018). This identifies opportunity to improve the capability on creativity and 
innovation by tackling on the issue of entrepreneurial creativity development. 

Design thinking is well recognized as the creative problem-solving process though human-centered 
approach. The process itself is a not linear with the strong focus on deeper understanding of users or customers 
to arrive with clear problem statement. The potential ideas emerge from different point of view brainstormed 
and transformed into prototypes for testing and retesting until they fully meet or exceed the customers’ 
expectations. The process simulates the life cycle of entrepreneurs and can foster creative thinking among 
learners(Linton & Klinton, 2019). 

Social media have been used to promote entrepreneurial education with variety of platforms (Line, 
WeChat, Facebook Messenger) all around the world due to its unique features to drive learners ‘engagement, 
experiences and collaborations to ultimately improve learning outcomes(Wu & Song, 2019). However, there are 
limited knowledge and studies on how to use social media learning to improve entrepreneurship or 
entrepreneurial creativity. Teepapai and Karawek (2018) found that the company’s culture and learning 
environment has a strong impact on creative ideas and innovative outputs of the employee; therefore, it is 
recommended to invest in the development of creativity by leveraging new connected technology that can 
promote learning among employees. 

Therefore, this article will explore on the literature of entrepreneurial creativity, design thinking and 
social media learning, as well as an in-depth focus group study with young managers in a multinational 
company. The article ends with recommending a learning design to promote young managers’ entrepreneurial 
creativity with design thinking in a social learning platform 


Corporate Entrepreneurship (Intrapreneurship) 

Successful innovation is one of the key success factors for companies to achieve business goals by 
growing revenue, profit margins, market share and corporate image(Miller & Bauer, 2017). Pinchot and Pellman 
(1985) are ones of the first inventors for “Intrapreneurship” (corporate entrepreneurship) by defining as the 
individual who has the vision above their responsibility with some creative ideas. Corporate entrepreneurship or 
intrapreneurship is another kind of entrepreneurship when employees drive the business with strong sense of 
ownership and will go extra mile to deliver success to the business with new creative ideas or innovations 
(Trifan, Guica, & Micu, 2012). Corporate entrepreneurs are the one who doesn’t have intention to create their 
own venture but willing to work within an organization with the strong sense of ownership. For an entrepreneur 
to become successful, he needs to have managers with corporate entrepreneurship quality to deliver 
innovation(De Lourdes Prado, MacHado, Mafra, & Maria Campos, 2012).There are 3 components of corporate 
entrepreneurship; proactiveness, creativeness and risk bearing (de Jong et al., 2011) cited in (Miller & Bauer, 
2017). Christensen (2011) cited in (Miller & Bauer, 2017) categorized 4 different type of creative entrepreneurs; 
(1) creative entrepreneurs are the one that introduce a new venture with differentiated products to capture unmet 
needs (2) creative intrapreneurs are the one who introduce a new business within a corporation (3) product 
creator are the one who creates new product or services (4) process creator are the one to continue to improve 
the efficiency or create new process. This reflects that there are 3 types of corporate entrepreneurship at 3 
different aspects from business within organization, product innovation and process improvement. They are all 
centered around creating or innovating to increase the organization’s competitive advantage. That’s why 
corporate entrepreneurship is identified of core competency in the corporate world, especially in international 
firms e.g. Unilever, P&G, Nestle’, and S.C. Johnsons & Sons, to generate innovation. Teltumbde (2006)said that 
corporate entrepreneur or intrapreneur can be more important than entrepreneur since an entrepreneur creates 
new venture while an intrapreneur work on it to success and achievement by adapting and reinventing the wheel 
based on the changes in business environment. 

Menzel (2007) cited in (Hanns C. Menzel, Aaltio, & Ulijn, 2007) explains the process of corporate 
entrepreneurship that has 2 levels; the organizational and individual level meaning the corporate entrepreneur 
can occur as an individual employee or as a group of employee. The procedure commences with opportunity 
recognition, exploitation and ends with creative solutions or innovations as in products, services, technology and 
processes as the figure below. 
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Fig. 1. Corporate entrepreneur process(H.C. Menzel, 2007) 


Entrepreneurial Creativity 

Creativity covers 2 dimensions; innovativeness and functional (Runco and Jaeger 2012) cited in (Liang 
et al., 2019). Innovativeness means being new, differentiated, and unexpected while functional means useful, 
handy and practical. Entrepreneurship requires creativity in making the new business venture innovative and 
functional for the market(Sternberg and Lubart,1999) cited in (Liang et al., 2019) Entrepreneurship and 
creativity are interrelated since creativity complements entrepreneurship in every stage from analyzing the 
opportunity gap in the market, think creatively on the potential solutions to address customer unmet needs, 
develop business plan and action plan to deliver the actual product to market. Creativity is considered to be a 
critical component of entrepreneurship in order to identify potential markets, create new ideas, and 
commercialize them(Saptono et al., 2019). In every stage, there are always uncertainties along the way which 
entrepreneurs or corporate entrepreneurs must employ creative problem-solving and innovative thinking to get 
through those situations. Entrepreneurial creativity was defined by Amabile (1997) cited in (Jing & Anja 
Svetina, 2014) as the creation and action of innovative business solutions or campaigns to launch new products 
or services. There are 2 key aspects of entrepreneurial creativity; creative problem-solving and creative 
innovation which are required skills in every process of entrepreneurship. Entrepreneurial creativity is not an 
inborn ability but can be developed from actual rea life situations. Researchers in the field of industrial 
psychology suggests nurturing an employee’s creativity is one of the critical success factors to enhance to one 
step ahead of competition as innovative organization(Ahlin, DrnovSek, & Hisrich, 2014). This is because a 
creative employee is proactive and look for opportunities to improve their way of working and anticipate the 
consequences for every step of actions. Price, Stoica, and Boncella (2013) investigates the relationship between 
innovation, knowledge and performance in family and nonfamily firms and found that innovation and creativity 
drive superior firm performance; therefore, it’s recommended to continue to come up with new product, 
services, processes to achieve business targets. 

Jing and Anja Svetina (2014) investigates entrepreneurial creativity with the resource-based view that 
entrepreneurial creativity happens during the gathering and allocation of resources under the context of resource 
constraints. There are 4 types of corporate innovations.(Trifan et al., 2012) 

1. Product or service innovation refers to the launch of new superior and differentiated products or 

services, comparing to the existing products in the market 

2. Process innovation refers to the application of a novel process to improve the efficiency and 

effectiveness through the introduction of new technology and way of working 

3. Marketing innovation refers to the launch of a novel marketing mix component of a product or 

services 

4. Organization innovation refers to the application of organization arrangement, re-organization, 

and new organization structure 

a 

In a corporation, each department or division can be managed like a venture which requires innovation 
to fuel the success; therefore, the organization required human resources with strong sense of entrepreneurial 
creativity. Trifan et al. (2012)mentions the 5 senses of corporate creator, explaining the behavior and attitude of 
employees with creativity 

1. Internal Sense describes the behavior that the individual is aware of the firm’s business 

performance, key issues and opportunities. The individual has a strong overall organizational 
network awareness and can get things done through collaborating with another functional team 

2. External Sense refers to the behavior that the individual knows the market insight and competitors 

so well that he’s able to analyze opportunities and threats that could happen in the future 
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3. Positional Sense describes the behavior that the individual can leverage strong competitive 
advantage of the company to generate new product ideas to achieve goals 

4. Strategic Sense refers to the behavior that the individual has strategic thinking capability to 
evaluate various business decision choices and recommend the most appropriate one for the 
company with strong justification 

5. Value Creation Sense describes the behavior that the individual has the strong sense of 
ownership, dedicated to creating value to the customers and the company by calculating on all 
risks before making or recommending decisions. 


Value 
Creation 
Sense 


Positional 
Sense 


Strategic 
Sense 


Fig. 2. 5 Senses of Corporate Innovators (Trifan et al., 2012) 


Design Thinking Process 

Design thinking has been used widely in the field of business, engineering and architecture to deeply 
understand the roots cause, define, redefine the problems to arrive with best-fit solutions through the process of 
prototyping and testing and now design thinking plays more role in education settings.(Koh, Chai, Wong, & 
Hong, 2015). Design thinking refers to a creative problem-solving method that’s systematic and collaborative by 
focusing on human-centered approach from empathizing (understanding), identifying problems, ideating 
possible solutions, prototyping from ideas to tangible solutions and testing with the target customers(Luchs, 
Swan, & Griffin, 2015). The process itself is iterative and nonlinear that help improve the quality of solutions to 
ensure it best fit the customer’s expectation. Design thinking will be best worked for the problem or opportunity 
that’s not well-recognized and ill-defined and this process has been used with proven success for new business 
creation, new product development and internal process improvement(Luchs et al., 2015). 

Jaitip promotes design thinking with research-based design namely: Digital Learning Design, Define 
Gap, Root cause analysis, Conceptualize, Learning Design, Rapid prototype, and Evaluation and Revision(Na- 
Songkla, 2018). Design thinking, originated from the research study of The Hasso Platter Institute of Design or 
D.school at Stamford University, refers to the thinking process to deeply understand the target groups from 
different perspectives to create new ideas, prototype and refine the output or process continuously and this 
process has been used to support instructional design to deliver learning outcomes among diverse learners 
fueling by technology. Design thinking process consists of 5 key steps; 

1. Empathize is the first step to understand in-depth of target group through interviewing, observing 
as well as collecting the past success and failures including participating with target groups’ 
activities. 

2. Define is the step to collect and synthesize actual data and facts to identify the root cause of the 
problem. 

3. Ideate is when people come in groups to brainstorm different ideas with tools to find solutions as 
many as possible to address the key root cause being analyzed in the prior step. 

4. Prototype is the step to transfer idea to become tangible product or service prototype and trigger 
discussion and criticism from the team and this step needs to happen very fast to see the flaws, 
weaknesses of the idea for further improvement. 

5. Testing is when to test the prototype with real customers to observe and evaluate the efficiency of 
the product and utilize the result to keep improving the prototype until it meets customers 
‘expectation. 
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Se PROTOTYPE 


Fig. 3. Design Thinking Process (Stanford d.school) 


Entrepreneur or corporate entrepreneur are daily facing with uncertainties with many ill-defined 
problems to cope with and survive in a competitive market environment; therefore, the life of them is very 
complex and nonlinear. In the past, the education for entrepreneurship is focusing on “about” which means 
teaching by focusing only the theoretical part of entrepreneurship while neglecting the “through” which is direct 
experiencing through the journey of entrepreneurship(Linton & Klinton, 2019). Design thinking process mirrors 
the key process of entrepreneurship and corporate entrepreneurship and the ultimate goal of design thinking 
process is to solve problem with creative ideas, products or services which is appropriate to nurture 
entrepreneurial creativity. Linton and Klinton (2019) investigated and found that a new teaching method with 
design thinking process can be effective approach to entrepreneurial education since the learners got a more 
practical experiences through experimenting in real life situation and this can help develop both skill and 
mindset of entrepreneurial creativity. Schiele and Chen (2018) study design thinking and digital marketing skills 
in marketing education and found that learners improved on key skills; understanding, creativity, 
communication skills, technology skills, critical thinking, and collaboration(Glen, Suciu, Baughn, & Anson, 
2015). 


Social Media Learning Platform 

Na-Songkla (2018) said that connectivism paradigm emphasizes the learning from the connection with 
information technology and groups of people. The social connection is the integral part of knowledge creation 
and the more social connection, the more knowledge being generated. This paradigm has a strong impact on our 
way of living, communication and learning. There are 4 aspects of connections; social media, social network, 
cloud-based learning and open education. 

Social media is the digital media as means for communication within social network by connection to 
the internet. The user interacts with each other by creating their own contents as well as consuming contents 
from others in the network and there are different types of social media for learning; (Na-Songkla, 2018) 

Blogs 

Collaborative Writing 
Video blog 

Photo Sharing 
Podcast 

Virtual Reality 
Crowd Sourcing 
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Wu and Song (2019) discusses the benefits of social media learning as follows; (1) enhance learner 
engagement, participation and experience (2) integrate outside resources as part of learning environment (3) 
create a group of application practice among learners. The social media enables the learners to interact with 
other learners and teachers in an open and collaborative environment; hence, increasing the rate of participation 
among learners. By nature, social media is ubiquitous in all aspects of our lives, from personal, family, work and 
education. Social media is proven to be key facilitator and driver of creativity(Bhimani, Mention, & Barlatier, 
2018). Social media is extensively used by entrepreneurs for marketing, business networking, information 
search, and crowd funding(Olanrewaju, Hossain, & Whiteside, 2019). 

Thailand has the highest social media penetration in Southeast Asia with 74% vs. total Southeast Asia 
at 61%, translating to more than 51 million people are actively using social media and they use through mobile 
device. The average time spent at 3 hours 11 minutes per day. The top 5 social media in Thailand are Facebook, 
Youtube, Line, Facebook Messenger and Instagram, accordingly. The majority 34% of social media users are 
25-34 years old.(source : we are social https://wearesocial.com) This identified that social media is the integral 
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part of young working adults for both work and personal life. Therefore, social media can be an appropriate 
platform to promote entrepreneurial creativity among young managers in Thailand. 

Social media for adult and higher education is getting higher attention for both content dissemination 
and evaluation and the research results demonstrated that there’s favorable consequences of social media in 
driving in-depth learning experience and the uplift of certain skills like collaboration and 
organization(Stathopoulou, Siamagka, & Christodoulides, 2019). Al-Rahmi, Alias, Othman, Marin, and Tur 
(2018) investigated a model of factors affecting learning performance using social media in Malaysia higher 
education and found that social media enhances collaborative learning and engagement though group 
assignments. Furthermore, O’Boyle (2014) studied mobilizing social media in sport management education by 
focusing on facebook and twitter as part of blended learning and concluded that social media delivered strong 
learning platform that can increase student and staff engagement. 


Young managers’ Insights on Development Opportunities 


Focus Group Discussion 

One focus group of 8 young managers in multinational company were conducted to understand the 
need gaps of existing entrepreneurial creativity development to propose the future recommendation. The 
respondents are between 28 to 35year-old manager, with 5-10 years of working experiences in multinational 
companies. The discussion guideline covers perceived entrepreneurial creativity skill, past development 
activities, needs for future entrepreneurial creativity development. 


Perceived Entrepreneurial Creativity 

In terms of perceived entrepreneurial creativity, most young managers rated themselves at the 
moderate level and struggled to improve their skill set since they get used to working under the close 
supervision and guidance from their superiors and regional team. In a multinational firm context, there’s always 
standard operating procedures (SOP) as patterns or ways of doing things and even template for each work to 
follow. This is because global and regional team will need to consolidate and compare between countries and 
regions; therefore, not allowed to deviate from the template or structure. If young managers are asked to work 
on things without template, they would find it difficult on how to start with. This can be called “Template 
Syndrome”. Most of them come from sales and marketing team and what they need to do on a yearly basis is 
creating the marketing plan. However, they got a template from global and regional team to follow rather than 
having a blank sheet paper to start with. They ended up filling forms rather than thinking analytically and 
creatively on what strategy should be proposed. 


Past Entrepreneurial Creativity Development 

Entrepreneurial creativity has been part of their functional competency for sales and marketing and 
they are all evaluated in the performance review on this perspective; however, they are not clear what it means 
to their role and haven’t got a chance to practice in the real settings. Most of the past development focused on 
theoretical framework in an e-learning module rather than real life experiential learning. This means that there’s 
limited collaborative working session with other learners. They are requited to complete the session by deadline; 
however, they don’t feel engaged with the materials since they mostly spend time alone with e-learning module 
and being tracked by human resources or training team. Training is more like a mandatory but not inspires them 
to develop themselves. 


Need for Future Development 

In terms of wish lists for future entrepreneurial creativity development, they are asking for a higher 
quality of development session by focusing on real-life case study where they can work with peers to learn how 
to transfer of knowledge to practice. They are not against e-learning; however, they think there should be mixed 
of e-learning and classroom where they can ask questions, discuss with trainers and peers. They are expecting a 
workshop session where they can even bring their real case at work to discuss with other learners and trainers in 
the workshop session. The e-learning platform shouldn’t be one-way learning and should be able to access 
anywhere any time since now it can only access via laptops with VPN, not mobile friendly. 

In conclusion, there’s potential to revamp the entrepreneurial creativity development by enhancing 
experiential learning as well as engagement among learners. The learning design should stimulate deep and 
collaborative learning in real working situation; hence, giving the opportunity to think of the current work and 
apply to improve on the day to day work. Engagement, experiential, collaboration and inspiration are keys to 
success. 
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In conclusions 


Promoting young managers’ entrepreneurial creativity in social learning platform 

From the literature and focus group discussion, I’d like to propose the learning design for 
entrepreneurial creativity by leveraging design thinking process to mimic the experiences of corporate 
entrepreneurship through empathize, define, ideate, prototype and test by using the case of new product 
development for market and sales function. Here’s the proposed learning design with a combination of design 
thinking process, cognitive tools and enabling technology to support experiential, collaborative, real case 
learning design to develop entrepreneurial creativity. The case study content can be based on the product 
category that the company is competing in or the expansion plan in the future so that the team can have the basic 
understanding of the category and target customers insights. 


Table 1 depicts the proposed learning design for entrepreneurial creativity 


Design Thinking Process — 
Learning Design 


Cognitive Tools 


Enabling Technology 


1. Empathize: 

- Interview and observe target 
groups to understand the customer 
journey and pain points for 
assigned product category 


- Interview & Observe guide 
- Customer Persona 
- User profile canvas 


Purpose: Interact with target 
customers 

- Line Video Call 

- Facebook Messenger 


2. Define: 
- Define the key unmet needs of 
target customers and root causes 


- Fishbone 
- WH questions (What, When, 
Where, Why How?) 


Purpose: Collectively define 
customers paint points 

- Line Notes 

- Line Albums 


3. Ideate: 

- Brainstorm with team as many as 
possible on solutions or product 
ideas 


- Brainstorming sessions 
- Idea communication sheet 
- Creatively techniques 


Purpose: Collaborative platform to 
gather ideas and discuss 
- Line Album and Notes 


4. Prototype: 
- Create product concept and 
initial prototype 


- Product Concept board 
- Boxing and shelfing 


Purpose: Platform to display ideas 
and prototype 
- Line Album and Notes 


5. Test: 

- Conduct a focus group interview 
to test with target customers to 
gain feedback and room for 
improvement 


- Conduct A/B testing 
- Focus group discussion 
- Use feedback-capture grid 


Purpose: Interactive forum to test 
ideas with target customers 

- Line Album 

- Line Messaging with groups of 
potential customers 

- Line Voting 

- Line Survey & Feedback 


This is the initial thought of potential learning design and will need to go through a rigorous study to 
validate what will work for young managers to enhance the effectiveness of the entrepreneurial creativity 
development. For the future research study, it’s recommended to 

1. Conduct a survey in a larger scale on need assessment of entrepreneurial creativity among young 


mangers 


2. Research and design the learning design, components, and technology for developing 


entrepreneurial creativity 


3. Test and validate the learning design, components, and technology among young managers to 


propose the most appropriate learning design and platform 
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Abstract 


Extension Agents across the country work to incorporate emerging technologies in their programming. 
Today’s youth have grown up with technology as a major component in their lives. 4-H Extension Agents are 
teaching youth using the 4-H slogan, “Learn by Doing.” The educational focus in 4-H is experiential learning 
(Borden, Perkins & Hawkey, 2014). However, Agents struggle to keep up with ever-changing technologies. A case 
study of Kentucky 4-H Extension Agents looked at technology use among Agents and how learner engagement is 
affected. The qualitative study included a focus group and questionnaire. Results showed technology is being used 
primarily in educational programming and marketing. Facebook and Instagram were the most used social media 
outlets. Barriers to technology use included the expense, lack of time by staff, and training needs. Results showed 
Agents feel technology use in programming promotes engagement if used properly. Tablets were listed as the most 
engaging technology, followed by laptops. 


Introduction 


“It is generally acknowledged that engagement plays a critical role in learning.” (D’ Mello, Dieterle, & 
Duckworth, 2017). “Efforts are made to enhance in-class learner engagement because it stimulates and enhances 
learning. However, it is not easy to quantify learner engagement.” (Alimoglu, Sarac, Alparslan, Karakas & 
Altintas, 2014). Extension programs, and especially 4-H programs are known to be hands-on in nature. The more 
recent push for STEM programs has allowed for even more incorporation of technology in 4-H programming. Just 
as many schools have transitioned to laptops or tablets for all students, a few 4-H programs have obtained funding to 
be able to offer laptops or tablets to all participants during educational programs. 

“Research shows that multifarious benefits occur when students are engaged in their own learning, 
including increased motivation and achievement. However, there is little agreement on a concrete definition and 
effective measurement of engagement.” (Sinatra, Heddy, & Lombardi, 2015). Fredricks, Blumenfeld, and Paris 
(2004) further break down engagement into behavioral engagement, emotional engagement, and cognitive 
engagement. 

Behavioral engagement includes actions such as following the rules and refraining from disruptive 
behaviors while emotional engagement includes the participants’ reactions and feelings toward the learning 
experience (Fredricks, et al., 2004; Fredricks, 2011). Cognitive engagement refers to the learner’s investment in 
learning (Fredricks, et al., 2004). There is overlap between the three areas of engagement. Fredricks, Bluemenfeld 
and Paris (2004) share types of individual needs: needs for relatedness, need for autonomy, and need for 
competence. These needs have been tested related to engagement. “Ultimately, although engagement might begin 
with liking or participating, it can result in commitment or investment and thus may be a key to diminishing student 
apathy and enhancing learning” (Fredricks et al., 2004, p. 82). Just as teachers love to see youth excited about 
learning, Extension Agents also love to have participants who are fully engaged and anxious to participate and learn. 
Schlechty (2011) defines five levels of student engagement: rebellion, retreatism, ritual compliance, strategic 
compliance, and engagement. In this order, the levels advance from least to most engaged. 


Methodology 


A case study was conducted to examine technology use among 4-H Extension Agents in Kentucky. The 
qualitative study included a focus group, as well as an open-ended questionnaire completed by five individual 4-H 
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Agents. The purpose of the study was to gather information about current technology use and barriers to technology 
use among 4-H Agents in Kentucky. 

The focus group was conducted at the District 5 4-H Agent’s Retreat. Thirteen 4-H Agents were present 
and participated in the open discussion. The entire discussion was recorded and notes were also taken during the 
focus group. 

An e-mail invitation was sent to the distribution list of Kentucky 4-H Agents inviting them to participate in 
the questionnaire related to technology use among 4-H Agents. Four Agents or specialists volunteered to 
participate. Two additional Agents were specifically asked to participate due to their use of technology in their 
county program by the suggestion of the Assistant Director for 4-H Youth Development. A total of five 
questionnaires were collected. Four were Agents who volunteered to participate and one was specifically asked to 
complete a questionnaire. 

Agents who completed questionnaires ranged in years of service as a 4-H Agent from 6 years to 26 years. 
The State 4-H Office, as well as Districts 1, 4, and 5 were represented. Three participants were female and two were 
male. Questionnaires were reviewed to look for common themes. Thematic coding was used and compared to the 
focus group discussion as well. 

The questionnaire included ten open-ended questions and one final question where Agents were asked to 
rank four technologies in regards to learner engagement. Agents were asked to identify the county in which they 
work, as well as their years of service. The following is a list of questions asked: 

1. What are the main ways you are currently using technology in your 4-H program? 

Are you using technology in marketing? Please provide examples. 

What other ways are you using technology in your 4-H program? Please list examples. 

In your opinion, what are the strengths of technology use in the Kentucky 4-H program? 

What do you see as opportunities for Kentucky 4-H in regards to technology? 

What do you feel are the weaknesses in regards to technology use in the Kentucky 4-H program? 

What do you feel are the barriers, if any, to technology use in your program? 

How do you feel Kentucky 4-H is perceived in the area of technology use? 

Do you feel the use of technology in educational programming affects learner engagement? Please 

explain. 

10. Can you provide examples from your own technology use in programming of how you feel it has 
affected learner engagement? 

11. Please rank the following items in terms of learner engagement. Rank #1 the technology you feel 
provides the most learner engagement and #4 to the least, ranking all four. 

Tablet (such as iPad, etc.) Projector Laptop Videoconferencing 


Me ON ON NS RS 


Results 


Kentucky 4-H Agents are using technology in a variety of ways. However, the most popular use of 
technology from this sample and focus group is in educational programming, followed by marketing. Most 4-H 
Agents are using a variety of social media outlets to market, as well as apps such as Remind. Social media use is 
most popular on FaceBook, but also Agents are using Instagram and Twitter. 

When asked in an open-ended question about how they use technology, all respondents stated they use 
technology in educational programming. Marketing was listed as the next way Agents use technology most, 
followed by communications. When asked how Agents use technology in marketing, all respondents noted the use 
of Facebook. The second most noted technology used in marketing was Instagram. 

When asked about the strengths of technology use in the Kentucky 4-H program, marketing and outreach 
are noted. Promotion of programs and interactions through social media are found to be strengths in Kentucky 4-H. 
Outreach to previously unreached audiences is also found to be a strength, as well as information outreach to teens. 

When looking at opportunities for technology use in the Kentucky 4-H program, the use of emerging 
technologies and providing more learning opportunities for youth are reported. Virtual training is also listed as an 
opportunity. 

Weaknesses of the use of technology in the Kentucky 4-H program include being slow to embrace changes 
in technology. There is a need for more curriculum developed with technology use in mind. The general feeling is 
that technology is not used enough and that more support is needed. 

When looking at barriers to technology use in the Kentucky 4-H program, expense and lack of access are 
the number one barriers. Followed by a lack of time by employees to learn to use technologies or create new 
programs. 
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The perception of Kentucky 4-H in the use of technology by those Agents surveyed shows a common 
feeling that Kentucky 4-H is weak or behind the times in the use of technology. However, it was noted that 
Kentucky 4-H is a front runner in use of the National Youth Science Day experiment. 

All Agents completing the questionnaire feel that technology contributes to learner engagement. Agents 
described incorporating technology as adding excitement to the program and improving the learning experience. 
One Agent noted that the variation in instructional design improves the experience and engagement. Another stated 
that new technologies must be carefully and deliberately worked into educational opportunities. Meeting youth 
where they are on the digital technology continuum was seen as a way to engage participants. 

Respondents were asked to rank four technologies in terms of learner engagement. The technology the 
respondent felt was the most engaging would have been rated as number one. The least engaging technology was 
rated with number four. Tablets such as iPads were ranked as the most engaging by four out of five respondents. 
Laptops were ranked as the second most engaging. Videoconferencing and use of a projector had the same 
combined scores and tied for third. 

The focus group included many of the same themes that were recognized in the Agent questionnaires. 
Social media was noted as a major use of technology by most Agents. Barriers included skill level and knowledge 
as well as different levels of resources in counties. Agents also noted expectations are that they automatically know 
how to use various technologies, which may not be the case. Opportunities noted included the use of more online 
training for Agents in order to cut costs associated with time and travel. 


Conclusion 


Extension Agents will continue to work to find ways to incorporate emerging technologies into their 
programs. A detailed study is needed to more clearly identify Agents’ technology needs in Kentucky. Program 
participants will benefit from engaging programs built on solid instructional design principles. Engagement is 
difficult to measure, however, Agents note programs with incorporated technology are of interest to youth. One 
Agent described the process as meeting youth where they are with technology in order to engage them. 
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